LForging  a Link 

B etween 

Accountability 

and 

Improvement  In  Schools-i 


jS  « 

= c_> 

3 

O T3 

S « C 

O O ro 

< £ - 
C o 
OJ  J)  L±j 


5 = 


ro 


at  o 

“ g -> 

— P ^ 

E C -O 

._  at 

_|  > T3 

o at 

re  2 ,-t; 

_,  Q.  T3 
c £ ? 

•5>m  t! 

j-  -o  a> 

a c 3 
y.  re  C3 


Contents 


^ i — f ^ 


3 


INTRODUCTION 
Lorna  Earl  and  Clay  Lafleur 


O ONTARIO  NOW 


-t-j 


4 


Joan  Green,  Lorna  Earl,  and  Richard  Jones 

A Report  from  the  Education  Quality  and  Accountability  Office 


O 

CD 

CO 


9 


Pauline  Laing 

Curriculum  Renewal  in  Ontario  Schools 


I Sandra  Sangster  and  Sylvia  Larter 

School  Board  Research  in  Ontario 


O 


<D 

25 

co 

28 


STANDARDS  & QUALITY 

Clay  Lafleur 

Curriculum  Reform 

Michele  Schmidt 

Historical  Perspective 

Bob  Wilson,  Joel  Weiss,  and  Graham  Orpwood 

What  Are  Standards?  Points  of  View  on  a Slippery  Concept 

Bruce  Brydges  and  Lorna  Earl 

No  Whining!  No  Gloating!  Large-Scale  Assessment 
as  a Starting  Point  for  Program  Changes 

Anne  Berens 

Performance-Based  Standards 

David  Ireland  and  Susan  Elliott 

Teacher  Moderation 


orbit,  Vol  28,  No  3,  1997 


Digitized  by  the  Internet  Archive 

in  2015 


https://archive.org/details/orbit1997v28i3oise 


32 

0 

3^ 

C 

39 

o 

•— 

42 

o 

44 

CL) 

CO 

49 

A DELICATE  BALANCE 
Michael  Kozlow 

Using  Data  for  School  Improvement 

John  Clipsham 

Longitudinal  Survey  of  Indicators  of  Success 


Paul  Favaro,  Judith  Fine,  Eva  Norman,  and  Anne  Bell 

Accountability  and  School-Based  Planning 

Connie  Kidd 

School  Improvement  Through  Self-Referenced  Student  Assessment 

Doug  Hamilton  and  Linda  Zaretsky 

Building  Professional  Communities  of  Inquiry  in  Schools 

Malcolm  Levin 

A Divergent  Viewpoint 


Orb 


o r 


A Commentary  on 
the  World  of  Education 
OISE/UT’s 
Magazine  for  Schools 

Volume  28,  Number  3,  1997 
Forging  a Link  Between 
Accountability  and  Improvement 
in  Schools 

Guest-edited  by  Lorna  Earl 
and  Clay  Lafleur 


Editor  of  Orbit:  Heather  Berkeley 
Editorial  Advisors:  Paula  Bourne, 

Amy  Casey,  Ardra  Cole,  George  Dei, 

Mark  Evans,  Lynne  Hannay,  Angela  Hildyard, 
Dave  Hunt,  Clare  Kosnik,  Peter  Rasokas, 
Howard  Russell,  and  Ruth  Scott 

Orbit  publishes  four  theme  issues  per  year  on 
topics  of  concern  and  relevance  for  teachers  and 
administrators  working  in  schools  and  school 
systems,  from  junior  kindergarten  to  the  end  of  high 
school.  Our  special  focus  is  Ontario.  Titles  for  ’97  are: 
Gender  and  Schooling;  Secondary  School  Change; 

Forging  a Link  Between  Accountability  and 
Improvement  in  Schools;  Literacy;  and  Assessment  and 
Evaluation.  Back  issues  are  also  available  on: 
Integrated  Curriculum;  Global  Citizenship;  Technology; 
and  School  Councils.  All  Orbit  issues  are  guest-edited 
by  experts  in  the  field. 

For  more  information,  contact  the  Editor 
Phone  416  923  6641,  ext.  2533  Fax:  416  975  1925 
E-mail:  hberkeley@oise.utoronto.ca 


OISE/UT 

252  Bloor  Street  West 
Toronto,  Ontario 
MSS  1V6 

Design:  Lori  Thompson/James  Ireland  Design  Inc. 
Printed  and  bound  in  Canada  by  Thistle  Printing 


Subscription  Rates 

$38.52  per  subscription  (4  issues  per  year) 
Inch  7%  GST.  Bulk  rates  available,  for  more 
information,  contact  ORBIT 
Phone:  (416)  267-2185  Fax:  (416)  265-7786 
Prepayment  is  requested.  Mail  cheque  or 
money  order  to: 

ORBIT 

2903  Kingston  Road 
Scarborough,  Ont., 

M1M  1N6 

Second  Class  Mail  Registration 
Number  4511 
ISSN  0030-4433  03 
Canadian  Publications 
Mail  Product  Agreement 
No.  638528 


N otes  from  the 


ccountability  is  a key  concern  for 
/ \ educators  both  in  Canada  and 
/ V abroad.  Increasingly,  we  are  being 
asked  by  our  publics  and  by  government 
for  proof  that  we  are  carrying  out  our 
mandate.  Do  our  students  have  access  to 
high-quality,  cost-efficient  education? 

How  do  we  measure  this?  And  how  do  we 
improve  results?  These  questions  are  being 
asked  internationally  as  public  education 
systems  undergo  massive  structural  and 
financial  reform. 

Obviously,  the  accountability  discussion 
in  education  is  not  simply  a matter  of 
statistics  but  of  gathering  and  reporting 
statistics  to  make  constructive  changes  for 
further  improvement.  The  Guest  Editors 
of  this  Orbit  theme  issue  urge  us  to 
approach  accountability  in  education  as 
an  “ongoing  inclusive  public  discourse” 
about  how  we  can  use  information 
to  make  changes  that  can  improve  and 
enhance  student  learning.  They  urge  us 
to  think  about  the  information  we  are 
collecting,  to  understand  the  standards  we 
are  applying,  and  then  to  reflect  pragmati- 
cally on  how  we  can  use  measurement 
information  to  make  a difference  to  our 
students’  achievement  and  progress. 


It  accountability  is  about  improvement, 
it’s  as  much  about  teachers  as  it  is  about 
students;  teachers  need  to  be  assessment- 
literate  to  work  practically  with  the  results 
of  test-scores,  whether  collected  locally 
or  internationally,  and  schools  need  to  be 
organized  so  that  teachers  can  plan 
changes  together  based  on  the  results.  This 
is  why  the  accountability  discussion  in  this 
Orbit  theme  issue  also  includes  a focus  on 
capacity-building  and  school  culture. 

The  key  to  progress  is  to  develop  a sys- 
tem of  external  standards,  and  to  establish 
means  for  teachers  at  the  classroom  and 
school  levels  to  act  on  this  information  to 
bring  about  improvements. 

The  present  Orbit  issue  helps  us  to 
understand  where  Ontario  is  now  with 
respect  to  curriculum  reform,  province- 
wide testing,  and  further  anticipated 
changes.  It  also  helps  us  to  link  the  infor- 
mation we  have  now  with  strategies  for 
school  improvement.  Lorna  Earl  and  Clay 
Lafleur  have  played  their  role  well  as  “criti- 
cal friends  of  Ontario  educators.”  Thanks! 

Michael  Fullan 
Dean,  OISE/UT 


o n 


Introducti 


aLorna  Earl 

Associate  Professor, 

Theory  and  Policy  Studies,  OISE/UT 
Formerly  Director  of  Assessment, 
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This  Orbit  edition  is  about  accountability,  but  that’s  not  the 
whole  story.  Although  performance  and  accountability 
are  watchwords  in  virtually  every  public  institution, 
accountability,  on  its  own,  is  a weak  and  powerless  concept.  In 
this  volume,  we  hope  to  draw  attention  to  the  fact  that  account- 
ability is  only  one  side  of  the  coin.  The  other  side  is  focused 
action  towards  improvement.  Together  they  are  a powerful  combi- 
nation; alone  they  are  poor  imitations  that  may  give  a fleeting 
illusion  of  change  but  will  soon  fade  into  oblivion. 

Accountability  is,  almost  by  definition,  an  emotionally  charged 
word.  It  usually  arises  when  there  is  discontent  and  anxiety  and 
it  is  often  associated  with  blaming,  divisiveness,  hostility  and 
polarized  positions.  At  the  same  time,  being  accountable  implies 
many  positive  actions,  things  like  responsibility,  quality,  striving 
for  excellence,  integrity,  trust,  and  confidence. 

Accountability  in  education  is  particularly 
important  because  the  impact  of  bad  decisions  in 
education  is  felt  directly  by  the  students  in  our 
schools.  The  articles  in  this  issue  highlight  some  of 
the  conceptual,  methodological,  social,  practical, 
political,  and  moral  issues  embedded  in  account- 
ability. Taken  together  they  describe  the  Ontario 
accountability  landscape,  draw  attention  to  some  of  the  various 
facets  of  standards  and  quality  in  education,  celebrate  some 
of  the  accountability  and  improvement  efforts  that  already  exist 
in  Ontario  school  boards  and  raise  some  questions  about  the 
consequences  of  going  down  the  accountability  road. 

As  researchers  who  have  worked  many  years  in  educational 
institutions,  we  have  learned  through  experience  that  doing 
research  on  assessment,  producing  reports,  and  publishing  statistics 
do  not  add  up  to  accountability.  We  have  a simple  caution  for 
the  reader.  We  have  become  obsessed  in  Canada  and  in  many 
other  countries  with  quantifying  the  products  of  education 
and  then  using  these  numbers  to  hold  people  to  account.  As 
researchers  we  certainly  support  continuing  to  do  research  and 
provide  data  but  we  need  to  position  the  resulting  information 
thoughtfully  and  wisely.  We  need  to  place  children  at  the  centre 
of  learning  and  strive  to  do  our  best  for  them. 

We  are  not  very  statistically  sophisticated  as  a culture  and  we 


are  prone  to  give  the  numbers  more  power  than  they  deserve; 
somehow  we  imagine  that  the  numbers  are  absolute  and 
immutable  instead  of  recognizing  that  they  are  simply  math- 
ematical representations  of  an  underlying  construct  with  some 
margin  of  uncertainty  in  their  interpretation.  Our  statistical 
illiteracy  creates  an  interesting  paradox.  We  are  coming  to  depend 
on  statistical  information  to  inform  decisions  even  though 
we  know  very  little  about  how  statistics  work  and  are  likely  to 
misunderstand  and  misuse  them. 

Even  if  the  statistics  are  understood,  just  having  information 
doesn’t  equal  accountability.  The  essence  of  accountability  is 
action,  not  reporting.  This  is  a fundamental  distinction  between 
accounting  and  accountability.  Accounting  is  gathering,  organizing 
and  reporting  information  that  describes  performance. 
Accountability  looks  forward;  it  is  the  conversation  that  follows 

an  ongoing  inclusive  public  discourse  about  what 
the  information  means,  about  how  it  fits  with 
everything  else  that  we  know,  and  about  how  to 
use  it  to  make  changes  that  can  enhance  and 
improve  student  learning.  This  kind  of  account- 
ability occurs  at  parent  nights,  in  town  hall 
meetings,  in  public  debates,  on  talk  shows,  in 
board  rooms  and  in  living  rooms  — wherever  people  meet  and 
talk  about  education.  And  the  conversation  is  likely  to  be  lively 
and  sometimes  emotional  because  the  future  for  our  children  is 
an  emotional  topic,  and  there  are  no  right  answers.  We  can  only 
hope  to  recognize,  consider,  and  explore  a whole  range  of 
perspectives  in  our  continuing  quest  for  an  educational  system 
that  is  both  excellent  and  adaptable. 

In  this  issue  of  Orbit  we  have  tried  to  present  a range  of  views 
from  educators  and  researchers  working  in  a number  of  different 
settings.  We  have  included  articles  from  the  EQAO,  the  Ministry 
of  Education  and  Training,  OISE/UT  and  other  universities, 
and  school  boards  throughout  the  province.  We  see  our  role  as 
“critical  friends”  of  Ontario  educators,  showing  you  the  work 
and  thinking  of  different  authors,  encouraging  you  to  stand  back 
and  reflect  on  their  ideas,  talk  about  them,  support  or  challenge 
them.  We  invite  you  to  enter  the  public  debate  as  an  informed 
and  constructive  participant.  Cl 


Accountability  without 
improvement  is  empty  rhetoric. 
Improvement  without 
accountability  is  whimsical 
action  without  direction. 
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How  can  the  public  be  reassured  that 
the  education  system  is  effectively, 
efficiently,  and  equitably  carrying  out  its 
mandate?  What  can  we  do  to  ensure  that 
relevant  information  is  collected  and  used 
to  improve  student  learning  and  education 
programs?  In  many  countries,  the  need 
for  answers  to  these  questions  has  resulted 
m increasing  public  pressure  for  changes 
m education  including  calls  for  more 
accountability  and  student  testing. 

In  Canada,  virtually  every  province/ 
territory  has  developed  new  standards  tor 
student  achievement  and  has  established 
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some  form  of  testing  program.  Some 
jurisdictions  conduct  relatively  low-stakes 
provincial  reviews  which  involve  school 
and  student  samples,  and  target  specific 
grades  and  subjects,  for  the  purposes  ol 
accountability  and  program  improvement; 
others  administer  both  provincial  reviews 
and  high-stakes  secondary  school  exit 
examinations  that  form  some  proportion 
of  students’  final  marks  in  specified 
senior-secondary  courses. 

A Brief  History 

In  addition  to  provincial  testing  initia- 
tives, all  provinces  participate  in  Canada’s 
national  testing  program  known  as  the 
School  Achievement  Indicators  Program 
(SAIP)  which  is  coordinated  by  the 
Council  of  Ministers  of  Education, 
Canada  (CMEC).  SAIP  involves  random- 
ly selected  provincial  and  national  samples 
of  13-  and  16-year-old  students.  Three 
subjects:  mathematics,  reading  and  writ- 
ing, and  science  are  assessed  on  a rotating 
basis,  one  subject  per  year,  and  data  are 
published  at  the  provincial  and  national 
levels. 

Many  provinces  also  take  part  in 
international  testing  programs,  such  as 
the  1995  Third  International  Math- 
ematics and  Science  Study  (TIMSS), 
that  are  conducted  by  organizations  such 
as  the  International  Association  for  the 
Evaluation  of  Educational  Achievement 
(IEA),  and  the  Organization  for 
Economic  Cooperation  and  Develop- 
ment (OECD). 

In  addition  to  testing  programs,  many 
provinces  have  established  so-called 
education  indicators  programs  to  provide 
their  publics  with  information  about  the 
quality  of  their  education  systems.  These 
programs  gather  important  data  and 
information  about  the  context  (commu- 
nity background  information),  inputs 
(information  about  financial  and  human 
resources),  processes  (methods  and  poli- 
cies to  achieve  outcomes),  and  results 
(education  outcomes)  of  the  education 
system.  The  resulting  annual  or  semi- 
annual reports  contain  information  that 


can  be  used  to  make  improvements  to 
education  systems  at  the  provincial  and 
local  levels,  and  to  help  jurisdictions 
become  better  communicators  with  their 
publics.  At  the  national  level,  CMEC, 
with  the  collaboration  of  the  provinces, 
has  developed  a Pan-Canadian  Education 
Indicators  Program.  While  recognizing 
that  m Canada  education  is  a provincial 
responsibility,  this  program  provides 
educators  and  the  public  with  statistics 
and  information  about  education  across 
the  country. 

a 


11 

During  the  past  three  decades  there 
has  been  little  provincially  initiated  assess- 
ment in  Ontario.  Prior  to  1 967  when  the 
program  was  discontinued,  there  was  a 
Grade  13  departmental  examination 
program,  involving  a variety  of  academic 
subjects,  that  was  used  to  screen  students 
for  university  entrance.  For  a brief 
period,  beginning  in  1968,  a series  of 
standardized  examinations  in  Grade  12 
subjects  such  as  physics,  mathematics, 
and  English  was  introduced.  In  its  first 
year,  testing  of  all  students  enrolled  in 
the  examinable  courses  was  mandatory; 
thereafter,  student  testing  was  conducted 
on  a voluntary  basis.  During  the  1970s 
and  1980s  there  was  no  provincial  testing 
program  although  some  school  boards 
elected  to  administer  norm-referenced, 
standardized  tests  to  students  at  certain 
grade  levels. 

In  the  mid-1980s  the  Ministry  of 
Education  began  a series  of  provincial 
reviews  in  subjects  such  as  mathematics, 
geography,  chemistry,  physics,  reading, 
and  writing  to  evaluate  the  effectiveness 
of  specific  programs.  These  reviews 
involved  samples  of  students,  and  were 
implemented  to  provide  information  for 
making  program  improvements.  For  two 
years,  beginning  in  the  1993/94  school 


year,  all  Grade  9 students  were  engaged 
in  a test  of  their  reading  and  writing 
abilities.  The  Province  generated  reports 
at  the  individual  student,  school,  school 
board,  and  provincial  levels,  and  it  was 
intended  that  the  information  in  these 
reports  would  help  the  people  involved 
reflect  on  their  learning  or  on  their  pro- 
grams in  order  to  make  improvements. 

Also  in  the  mid-1980s  the  Ontario 
Ministry  of  Education  began  a teacher 
in-service  and  examination  review  pro- 
gram called  the  Ontario  Academic 


Course  Teacher  In-service  Program 
(OAC-TIP)  to  ensure  greater  consistency 
in  how  student  marks  at  the  OAC  (final 
secondary)  level  were  determined  and 
awarded  across  the  province.  Generally, 
the  program  involved  providing  training 
to  OAC  teachers  on  how  their  students 
should  be  assessed  and  evaluated  accord- 
ing to  Ministry  guidelines.  After  an 
implementation  period,  examples  of 
teachers’  examinations  and  student 
responses  were  reviewed,  and  school 
reports,  indicating  the  degree  of  compli- 
ance with  the  guidelines,  were  produced. 
Approximately  a dozen  reviews  were 
conducted  before  the  program  was  sus- 
pended in  1995  to  await  anticipated 
changes  resulting  from  secondary  school 
reform,  and  pending  a review  of  the 
program  itself. 

Setting  up  EQAO 

The  impetus  for  establishing  the 
Education  Quality  and  Accountability 
Office  (EQAO)  had  its  origins  with 
reports  of  the  Provincial  Auditor  and  the 
Royal  Commission  on  Learning.  In 
his  1993  report  the  Provincial  Auditor 
recommended  that  "the  Ministry,  in 
conjunction  with  its  Commission 
on  Learmng,  should  study  the  entire 


During  the  past  three  decades  there  has  been  little  provincially 
initiated  assessment  in  Ontario. 


a 

Schools  may  not  be  prime  suspects,  but  their  past  reporting 
practices  have  not  improved  public  confidence. 


accountability  structure  with  a view 
to  improving,  province-wide,  the 
consistency  and  quality  of  education. 

“As  a minimum,”  the  recommenda- 
tions continued,  “the  Ministry  should 
increase  the  scope  of  its  monitoring 
mechanisms  to  better  assess  the  extent  to 
which  Ministry-developed  curriculum  is 
taught  and  the  expected  learning  out- 
comes are  being  achieved”  (p.76). 

Following  nearly  two  years  of  consulta- 
tions involving  students,  parents,  educators, 
and  the  wider  community,  Ontario’s  Royal 
Commission  on  Learning  (1994)  made 
numerous  recommendations  to  improve 
the  education  system.  In  the  words  of  the 
Commission,  “the  public  school  system  is 
responsible  to  the  public,  and  owes  it  to 
the  public  to  demonstrate  how  well  it’s 
doing  with  our  children”  (p.35). 

Consequently,  among  its  recommenda- 
tions, the  Commission  called  for  the 
creation  of  an  independent  agency  to 
monitor  the  effectiveness  of  the  educa- 
tion system  and  to  make  recommenda- 
tions for  improvement.  In  addition,  the 
Royal  Commission  said  that  the  new 
agency  should  administer  universal  testing 
at  Grade  3 (literacy  and  numeracy)  and 
Grade  1 1 (literacy),  and  there  should  be 
more  regular  assessment  of  other  courses 
and  programs  across  the  province. 

The  provincial  government  of  the  day, 
acting  on  the  recommendations  of  both 
the  Provincial  Auditor  and  the  Royal 
Commission  on  Learning,  appointed  Joan 
Green  (then,  the  Director  of  Education  of 
the  Toronto  School  Board)  Chief  Executive 
Officer  (CEO)  of  the  new  agency  which 
was  to  be  titled  the  Education  Quality  and 
Accountability  Office  (EQAO).  Green’s 
appointment  was  effective  May,  1995.  Soon 
afterward,  a new  provincial  government 
affirmed  its  support  for  EQAO  and  began 
preparing  documentation  for  legislative 
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approval  of  the  agency.  Bill  30,  the  legisla- 
tion to  formally  establish  EQAO,  received 
third  and  final  reading,  and  was  passed 
unanimously,  on  June  30th,  1996. 

The  Philosophy  of  EQAO 

In  the  agency’s  early  days,  the  CEO 
assembled  a committee  of  Ontario 
assessment  experts  to  research  and  to 
identify  assessment  practices  in  use  across 
Canada  and  around  the  world.  This 
research  was  conducted  in  recognition 
that  “Expertise  in  assessment  techniques 
and  effective  evaluation  designs  exists 
here  and  elsewhere,  and  EQAO  will 
draw  on  the  expertise  heavily,  in  the 
interest  of  creating  clear,  sensible  and 
coherent  measures  of  learning”  (Green, 
1996b,  p.59).  In  particular,  the  committee 
looked  at  the  relative  success  of  those 
approaches,  together  with  their  intended 
and  unintended  outcomes,  and  consid- 
ered whether/how  the  various  practices 
might  be  applied  in  Ontario.  The  design 
of  Ontario’s  current  provincial  assess- 
ments owe  much  to  the  work  of  this 
original  committee,  known  as  the  Design 
Options  Team. 

An  important  outgrowth  of  this  com- 
mittee’s work  was  that  EQAO  adopted 
a philosophy  that  gave  equal  weight  to 
its  two  principal  mandates:  improvement 
and  accountability.  Regarding  improve- 
ment, the  CEO  observed  that  “if  our 
students... must  be  both  literate  and 
numerate... they  must  be  creative,  adapt- 
able and  able  to  co-operate  with  others 
to  solve  problems... they  will  need  to 
be  life-long  learners”  (Green,  1996c). 

Perhaps  the  best  description  of  how 
improving  performance  can  be  attained 
was  offered  by  an  Ontario  Grade  3 
student  who  stated  that,  “Kids  need  to 
know  when  they’re  doing  good  work 
and  what  happens  next.” 


Provincial  Assessment 

During  the  winter  of  1996,  the  CEO 
and  members  of  her  assessment  staff 
conducted  province-wide  consultations 
with  Grade  3 and  Grade  6 teachers 
to  assist  the  office  with  the  design  and 
development  of  the  assessments. 

In  the  spring  of  1997,  EQAO  con- 
ducted a province-wide  assessment,  in 
both  French  and  English,  of  Grade  3 
children’s  mathematics,  reading,  and  writ- 
ing knowledge  and  skills.  Every  Grade  3 
student  participated  in  the  assessment. 

The  assessment  was  curriculum-  and  cri- 
terion-referenced, based  on  an  integrated 
unit  of  work,  built  around  a single  theme 
involving  reading,  writing,  and  mathe- 
matics activities  and  tasks,  and  took 
approximately  ten  days  of  classroom  time 
to  complete.  The  purpose  of  the  assess- 
ment was  to  provide  all  concerned  — 
students,  parents,  teachers,  and  the  public 
— with  information  about  students’ 
knowledge  and  skills. 

Individual  student  reports  were 
returned  to  the  students  and  schools 
before  the  end  of  June;  school,  school 
board,  and  provincial  reports  were 
prepared  for  release  in  October  of  1997. 
Information  gathered  through  the  assess- 
ment showed  where  students  were  doing 
well  and  where  they  were  having  difficul- 
ties. This  information  is  helping  teachers 
improve  curriculum  and  instruction  for 
their  students.  It  helped  parents  become 
aware  of  the  content  taught  and  standards 
expected  in  Ontario  schools,  it  provided 
a common  language  for  talking  about 
teaching  and  learning,  and  it  provided  the 
public  with  information  about  the  quality 
of  student  performance. 

In  addition  to  the  Grade  3 test,  a sam- 
ple of  Grade  6 students  were  assessed  in 
mathematics  in  the  spring  of  1997. The 
Grade  6 assessment  was  curriculum-  and 
criterion-referenced,  and  involved  mathe- 
matics activities  and  tasks  that  were 
conducted  over  eight  40-minute  blocks  of 
time.  The  purpose  of  this  assessment  was 
to  provide  a provincial  picture  of  students’ 
mathematics  knowledge  and  skills  in  order 
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to  improve  curricular  and  instruc- 
tional programs  and  to  enhance  public 
accountability.  Reporting  was  at  the 
school  and  provincial  levels  only,  and 
those  reports  were  released  in  October 
of  1997. 

All  teachers  administering  the  assess- 
ments, as  well  as  the  principals  ot  all 
of  the  province’s  elementary  schools, 
received  training  in  the  assessments.  In 
addition  to  the  achievement  information 
provided  by  the  assessment,  parents 
were  asked  to  complete  a home  ques- 
tionnaire about  their  child’s  reading, 
writing,  and  mathematics  habits,  and 
the  ways  in  which  they  support  student 
learning  at  home;  teachers  also  complet- 
ed a questionnaire  that  provided  infor- 
mation about  demographics,  classroom 
organization,  and  curriculum  and 
teaching  practices.  Analysis  of  these 
data  will  allow  EQAO  to  study  the 
relationships  between  and  among  various 
background  variables  and  student  perfor- 
mance; it  will  also  provide  valuable 
contextual  information  to  assist  the 
interpretation  of  results. 

It  is  intended  that  the  Grade  3,  all- 
student assessment  will  be  conducted 
annually;  development  of  an  annual, 
universal  high  school  “literacy  assess- 
ment” will  begin  when  the  new  sec- 
ondary school  program,  moving  from 
five  to  four  years,  is  established  (the 
scope  of  “literacy  assessment”  is  yet 
to  be  determined  — there  could  be 
more  than  one  type  of  literacy  involved); 
and  assessments  of  random  samples  of 
Grade  6 and  9 students  will  be  reviewed 
on  a regular  schedule. 

National  & 

International  Assessments 

In  addition  to  its  provincial  assessment 
program,  Ontario  participates  in  national 
and  selected  international  assessments, 
and  EQAO  manages  the  province’s 
involvement  in  these  projects.  Like  all 
of  the  provinces  and  territories,  Ontario 
participates  in  the  SAIP,  an  annual, 
national  test  of  randomly  selected 


samples  of  13-  and  16-year-olds  in 
either  mathematics,  reading  and  writing, 
or  science,  coordinated  by  the  CMEC. 
Assessments  have  been/will  be  conduct- 
ed in  the  spring  each  year,  according  to 
the  following  schedule:  mathematics, 
1993;  reading  and  writing,  1994;  science, 
1996;  mathematics,  1997;  reading  and 
writing,  1998;  science,  1999.  Reports  are 
released  at  the  provincial  and  national 
levels  only. 

In  recent  years,  Ontario  has  also  taken 
part  in  international  assessments  such  as 
the  Second  International  Mathematics 
and  Science  Studies  (SIMS  and  SISS), 
The  Second  International  Assessment 
of  Education  Progress  (IAEP  11)  in 
Mathematics  and  Science,  and  the  Third 
International  Mathematics  and  Science 
Study  (TIMSS).  Numerous  international 
studies  are  either  underway  or  are 
planned,  and  it  is  EQAO’s  responsibility 
to  make  recommendations  to  the 
Ministry  regarding  Ontario’s  involve- 
ment in  such  projects,  and  to  manage 
that  participation  should  involvement 
be  deemed  valuable.  Generally,  interna- 
tional reports  contain  international 
and  national  data,  and  it  is  up  to  the 
Canadian  coordinating  body  and  EQAO 
to  generate  Canadian  and  Ontario 
reports,  respectively.  Like  the  national 
testing  program,  information  from 
international  studies  permits  compar- 
isons of  educational  achievement, 
and  describes  how  countries  differ  in 
what  and  how  they  teach. 

In  Ontario,  all  assessments  are  con- 
ducted in  both  French  and  English.  As 
a follow-up  to  national  and  international 
assessments,  EQAO  conducts  research 
and  makes  recommendations  for  educa- 
tional improvements. 

u 


Education  Indicators 

EQAO  will  design  and  implement  an 
education  quality  indicators  program  that 
will  provide  information  to  support  plan- 
ning and  decision  making  at  the  school, 
school  board,  and  provincial  levels.  It  is 
expected  that,  eventually,  EQAO  will 
publish  regular  indicator  reports  that  will 
provide  information  related  to  the  quality 
ot  the  education  system  provincially.  By 
making  this  and  other  information  avail- 
able at  the  local  level,  it  is  anticipated  this 
program  will  assist  school  boards  to  mon- 
itor their  programs  more  effectively  and 
to  become  better  communicators  with 
their  own  publics,  thereby  enhancing 
accountability. 

In  addition  to  assessment  and  educa- 
tion indicators  programs,  EQAO  will 
design  and  implement  other  programs  as 
required  to  monitor  and  report  on  the 
effectiveness  of  education  in  the  province. 

Research,  Advice,  & Reporting 

EQAO  promotes  and  conducts  research 
m best  practices  in  assessment  and 
accountability.  For  example,  as  a follow- 
up to  the  TIMSS  study,  EQAO,  in  collab- 
oration with  the  Ontario  Association 
of  Deans  of  Education  (representing 
all  of  Ontario’s  faculties  of  education), 
conducted  comparative  research  on 
curriculum,  teaching  practices,  teacher 
education,  francophone  issues,  and  vari- 
ables related  to  student  performance. 

By  conducting  this  type  of  research  and 
making  the  results  available  to  the 
relevant  stakeholders,  the  information  can 
be  used  to  improve  education  programs. 

The  office  publishes  an  annual  report 
to  the  public  and  the  Minister  of 
Education  and  Training,  on  the  quality 
of  the  education  system.  This  document 


Expertise  in  assessment  techniques  and  effective  evaluation 
designs  exists  here  and  elsewhere,  and  EQAO  will  draw  on  the 
expertise  heavily,  in  the  interest  of  creating  clear,  sensible 
and  coherent  measures  of  learning. 


u 

Mistrust  of  the  old  "standardized"  tests  was  justified,  but  the 
new  approaches  to  testing  offer  positive  developments. 

V) 


provides  information  and  recommenda- 
tions in  a straightforward  manner  that 
will  be  of  value  to  educators  and  the 
wider  community. 

Characteristics  of 
EQAO  Assessments 

The  following  characteristics  of  EQAO  s 
assessments  originate  from  the  agency’s 
worldwide  survey  of  assessment  practices 
and  its  consideration  of  their  appropriate- 
ness in  the  Ontario  context.  The  assess- 
ments: 

• are  based  on  the  Provincial  Ministry 
standards  and  curriculum  that  indicate 
the  knowledge  and  skills  that  students 
should  acquire 

• are  performance-based  in  that  they: 

— are  made  up  of  tasks  that  continue 
during  class  periods  over  several  days 

— give  us  a broad  view  of  students’ 
knowledge  and  skills 

— ask  students  to  work  independently 
to  complete  challenging  tasks  in  every 
area  of  the  curriculum  being  assessed 

— require  students  to  illustrate  their 
thinking  processes  as  they  solve  prob- 
lems, express  opinions,  and  formulate 
answers 

• are  embedded  in  instruction  and  are 
often  integrated  across  subject  areas 

• allow  for  appropriate  accommodations 
and  exemptions  of  students  (where 
necessary,  students  with  special 
needs  are  accommodated  without 
compromising  test  validity:  in  rare 
cases,  students  can  be  partially  or  fully 
exempted  from  participating) 

• build  teacher  capacity  by: 

— modeling  exemplary  instruction 
and  assessment  practices 

— using  a range  of  authentic  resources 

— providing  teacher  training. 

• recognize  teachers  as  important  to 


program  success  (although  EQAO 
employs  professional  expertise  for 
test  design,  statistical  analysis,  and 
other  operational  elements  of  its 
assessment  programs,  it  recognizes 
that  teachers  are  crucial  to  the 
agency’s  assessment  and  accountability 
philosophy  and  involves  them  in  all 
key  aspects  ot  the  assessments  — 
development,  training,  administration, 
scoring,  and  reporting. 

Conclusion 

As  the  educational  world  faces  piercing 
scrutiny,  and  attempts  to  deal  with 
what  seem  to  be  insatiable  demands 
and  great  intolerance  for  ambiguity,  the 
obvious  erosion  of  trust  in  public  institu- 
tions is  becoming  widespread.  Schools 
may  not  be  prime  suspects,  but  their  past 
reporting  practices  have  not  improved 
public  confidence.  People  see  standards 
as  inconsistent,  reporting  as  unclear  and 
defensive.  They  seek  direct  evidence  of 
student  learning  as  the  product  for 
which  schools  are  to  be  held  accountable. 
The  public’s  scepticism  is  fueled  by 
media  reports  regarding  our  students’ 
comparative  performance  on  provincial, 
national,  and  international  tests.  The  pub- 
lic wants  impartial,  arm’s-length  reporting 
on  learning  standards  and  specific  refer- 
ence points  that  can  be  used  when  judg- 
ing school  results,  taking  into  account 
local  context;  the  public  wants  teachers 
to  be  competent  and  open  in  assessment 
and  reporting;  and  it  wants  reports 
that  make  practical  recommendations 
that  will  point  the  way  for  making 
improvements. 

Given  the  public  scepticism,  large-scale 
assessments,  done  impartially,  can  bring 
corroboration  to  our  work.  Mistrust  of 
the  old  “standardized”  tests  was  justified. 


but  the  new  approaches  to  testing  offer 
positive  developments.  They  can  bring 
more  stability  and  professionalism  to 
our  schools,  and  an  informed  and  power- 
ful response  to  the  demand  for  account- 
ability. The  new  standards  and  assessments 
can  eventually  improve  teaching  because 
they  depend  upon  trained  teacher  judg- 
ment. They  can  help  teachers  with  their 
work  rather  than  intrude  upon  it.  Public 
confidence  in  results  and  standards  will 
strengthen  respect  for  the  public  educa- 
tion system  (Green,  1996b).  El 
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Curriculum  Renewal 


Ministry  of 

Education  and  Training 


The  Curriculum  Learning  and  Teaching  Branch  at  the 

Ministry  of  Education  and  Training  has  several  important 
roles  in  the  renewal  of  curriculum  for  elementary  and 
secondary  schools  in  Ontario.  First,  the  Branch  develops  curricu- 
lum policy.  Based  on  overall  government  policy,  widespread 
input  and  professional  expertise,  we  outline  the  expectations  for 
learning  for  all  students.  With  a focus  on  knowledge  and  skills, 
and  an  understanding  that  sound  attitudes  must  be  transmitted  in 
the  instructional  process,  we  provide  curriculum  expectations  for 
students  in  our  schools. 

Second,  we  provide  a framework  for  standards  and  assessment, 
both  day-to-day  assessment  in  the  classrooms  and  province- 
wide assessment  carried  out  by  the  Education  Quality  and 
Accountability  Office.  We  have  already  begun  the  renewal  of 
elementary  curriculum,  and  have  decided  on  a four-level 
achievement  scale  for  subjects  in  this  panel.  Using  four 
Levels  of  Achievement  in  our  curriculum  documents, 
we  identify  for  teachers  the  general  criteria  which 
must  be  common  across  the  province  as  they  assess 
the  achievement  of  their  students.  It  is  these  Levels 
upon  which  instruction,  assessment,  and  provin- 
cial testing  are  based.  In  addition  to  developing 
the  framework  for  assessment,  we  work  to 
improve  the  curriculum  based  on  the  results 
of  province-wide,  national  or  international 
testing. 

Third,  we  develop  the  Standard 
Provincial  Report  Card  which 


provides  the  formal  link  between  teachers  and  all  parents. 

The  Report  Card  brings  together  curriculum  expectations 
and  assessment  scales  in  a manner  that  is  consistent  across  the 
province.  Although  we  realize  that  a report  card  is  only  one  part 
of  communication  about  student  achievement,  we  know  that  it 
is  a communication  likely  to  be  read  in  detail  by  parents  and 
students  alike,  and  that  it  must  reflect  our  belief  that  assessment 
is  curriculum-based  and  ongoing. 

Fourth,  in  partnership  with  the  Ministry  District  Offices, 
we  provide  initial  awareness  of  new  curriculum  to  boards  across 
the  province,  and  encourage  the  development  of  professional 
expertise  to  implement  new  curriculum.  Increasingly,  as  the 
College  ofTeachers  furthers  its  work  on  standards  of  practice 
for  the  profession  and  professional  learning,  we  will  work 
in  partnership  with  the  College  to  assure  co-ordmation 
between  the  curriculum  mandate  and  pre-service  and 
in-service  development  of  teachers. 

Our  most  recent  work  has  concentrated  on  the 
development  of  the  Ontario  Curriculum  for  Grades  1 
to  8,  and  the  accompanying  Standard  Provincial 
Report  Card.  By  describing  some  of  the  issues 
which  we  have  encountered  in  this  process,  we 
can  illustrate  our  ongoing  work. 

The  Report  Card,  for  example,  has  raised 
questions  about  how  assessment  of  student 
achievement  can  be  made  an  integral  part 
of  day-to-day  instruction  in  the  class- 
rooms of  Ontario.  The  new  report  card 


has  helped  us  all  think  about  the  ongoing  process  of  assessment 
which  contributes  to  learning  and  which  makes  helpful  commu- 
nication to  parents  possible.  We  pictured  the  teacher  sitting  down 
to  write  the  report  card,  and  knew  that  weeks  of  careful  work 
would  stand  behind  every  comment. 

Knowing  and  understanding  the  curriculum  expectations 
was  the  foundation  for  us.  Developing  the  same  understanding 
is  the  first  step  for  teachers  and  parents.  We  hope  that  all  schools 
will  take  time  yearly  to  explain  to  parents  what  the  curriculum 
means  for  their  children,  and  encourage  them  to  provide  enthu- 
siastic support  at  home.  When  parents  understand  the  expecta- 
tions, they  are  much  more  likely  to  understand  the  assessment 
and  have  confidence  that  learning  is  systematic,  relevant  and 
based  on  improvement. 

The  new  Ontario  Curriculum  is  more  specific  in  its  expecta- 
tions than  previous  curriculum. We  are  asking  teachers  to  assess 
learning  with  regard  to  specific  requirements,  and  give  parents 
very  clear  information  about  whether  students  have  the  stated 
skills.  The  Expectations  are  the  mandate  for  learning  and,  what- 
ever the  child’s  skills  or  challenges,  parents  need  to  know  where 
their  child  stands  with  respect  to  the  Expectations. 

We  thought  about  real-life  situations  and  how  they  might 
look  and  feel  when  teachers  encountered  them.  If,  for  example, 
a student  couldn’t  read  more  than  a simple  sentence,  it  would 
be  important  that  parents  have  that  information.  In  conferences, 
teachers  can  give  concrete  examples:  “This  is  a sentence  your 
child  can  read,”  the  teacher  could  indicate.  Then  she  could  add, 
“This  is  the  type  of  book  that  we  want  children  in  Grade  2 
to  be  able  to  read.”  The  information  is  not  a judgment  about  the 
worth  of  the  child,  and  teachers  must  make  that  point  very  clear. 
It  is  a statement  about  whether  the  child  has  learned  something 
that  we  think  is  valuable  to  know,  or  has  skills  we  think  it  is 
useful  to  possess. 

We  also  thought  about  the  range  of  assessment  strategies 
that  teachers  might  use.  We  use  the  word  “demonstrate”  in  the 
report  card  descriptions  because  we  believe  that  teachers  must 

be  looking  for  evidence  of  learning  in  a variety  of  different 
ways.  Tests  and  traditional  assignments  will  always  be  part  of 
the  repertoire,  but  we  were  very  clear  in  our  own  minds 
that  a range  of  strategies  is  essential.  We  know  how  suc- 
cessful portfolios  have  been  in  many  situations  and 
wanted  to  make  sure  that  all  kinds  of  evidence,  care- 
fully gathered,  would  create  a full  picture  of  student 
achievement. 

These  are  complex  times  and  another  issue 
we  faced  was  combining  different  types  of 
indicators  of  achievement.  We  had  clear 
requests  from  parents  and  many  teachers  for 
traditional  indicators  of  student  achieve- 
ment (marks  and  letters).  At  the  same 
time,  they  told  us  that  newer  approaches 


(levels)  were  definitely  worth  some  attention. 

The  framework  of  levels  we  have  provided  is  designed  to 
guide  teacher  observation  of  student  growth  in  each  subject,  and 
is  very  much  linked  to  curriculum  Expectations.  The  levels  are 
based  on  some  fundamental  factors:  independence  in  learning, 
breadth  and  depth  of  knowledge  and  skills,  accuracy  in  using 
skills  and  knowledge,  and  application  of  skills  and  knowledge  to 
new  situations.  If  teachers  keep  in  mind  these  guiding  principles, 
they  can  design  “rubrics”  or  more  detailed  descriptions  for 
specific  learning  tasks  in  which  students  are  engaged. 

“Rubrics,”  we  believe,  are  very  important  to  enrich  classroom 
learning  and  help  both  students  and  parents  understand  assess- 
ment. Rubrics  also  help  students  become  involved  in  the 
all-important  process  of  self-assessment.  There  are  some  simple 
examples  that  remind  us  how  important  self-assessment  really 
is.  Every  employer  knows  that  the  best  employee  is  the  one  that 
doesn’t  wait  to  be  told  whether  his  or  her  work  can  be 
improved.  The  stellar  employee  is  the  person  who  reflects  on 
work  done,  establishes  some  criteria  for  improvement  and  moves 
on  to  better  performance.  This  ideal  employee  proceeds  indepen- 
dently. Developing  clear  descriptions  of  student  performance 
(rubrics)  will  build  this  capacity  to  look  at  work  and  recognize 
when  and  how  improvement  can  be  achieved. 

In  dialogue  with  teachers,  the  ministry  will  be  exploring 
how  new  modes  of  assessment,  like  levels  of  performance,  can 
be  made  meaningful  alongside  marks  and  letter  grades.  We  realize 
that  report  cards  are  designed  to  communicate  with  parents, 
and  that  they  must  include  the  symbols  that  parents  tell  us  are 
meaningful  to  them.  At  the  same  time,  the  educator’s  role  will 
include  explaining  that  although  “B”  and  Level  3 have  a general 
correspondence,  different  approaches  to  assessment  accomplish 
different  things. 

Although  most  of  this  article  has  focussed  on  our  work  with 
elementary  renewal,  it  is  clear  that  the  same  cluster  of  roles 
will  be  part  of  secondary  development.  There,  too,  we  will  be 
responsible  for  beginning  with  the  development  of  curriculum 

Expectations.  Then  we  will  need  to  provide  a framework  for 
assessing  student  learning  and  a report  card  for  communi- 
cating that  learning.  Finally,  and  in  an  ongoing  way,  we 
will  need  to  encourage  professional  dialogue  in  Ontario 
about  how  the  continuing  connections  between  cur- 
riculum and  assessment  are  best  maintained  in  support 
of  learning  for  our  students. 

When  I set  out  to  write  this  article,  someone 
advised  me  to  think  about  why  I was  spending 
the  last  years  of  my  educational  career  engaged 
in  the  work  of  the  Curriculum,  Learning  and 
Teaching  Branch.  Having  set  forth  the  evi- 
dence, I will  summarize:  I like  work  that  is 
complex,  full  of  possibilities  and  decisions, 
and  of  genuine  benefit  to  learning.  Cl 


orbit,  Vol  2 8,  No  3,  1997 


School  board 

rese  a 


Sandra  Sangster,  President 
Sylvia  Larter,  Past  President 
The  Association  of  Educational 
Research  Officers  of  Ontario 


Association  of  Educational  Research  Officers 
of  Ontario  (AERO)  was  established  in  1972, 
primarily  for  the  purpose  of  providing  members  with  a 
forum  for  professional  development  and  networking 
within  what  was,  at  the  time,  a small  educational  research 
community.  Over  the  years,  the  membership  has  grown 
as  the  number  of  boards  with  Research  and  Assessment/ 
Evaluation  Departments  has  increased.  In  addition,  the 
membership  has  broadened  beyond  school  board 
researchers  to  include  members  employed  in  universities, 
test  publishing  centres,  the  Education  Quality  and 
Accountability  Office,  and  the  Ministry  of  Education 
and  Training. 

The  roles  which  AERO  members  fulfill  are  as 
diverse  as  the  school  boards  in  which  they  work.  These 
range  from  a focus  on  school  and  classroom  based 
research  to  support  for  system-wide  assessment  and 
public  reporting  and  accountability  initiatives. 

Assessment 

A particularly  critical  area  of  work  for  many  AERO 
researchers  in  the  past  ten  years  has  been  the  concep- 


tualization and  development  of  content  and  performance 
standards  and  methods  for  measuring  student  achievement 
with  reference  to  these  standards.  These  efforts  focused 
first  on  the  areas  of  language  and  mathematics,  but  are 
now  extending  to  other  subjects  such  as  science. 

AERO  members  have  been  central  contributors  both 
in  designing  assessments  and  ensuring  that  approaches  to 
assessment  meet  a variety  of  purposes,  ranging  from 
daily  classroom  observation  and  assessment  to  account- 
ability at  the  school  and  board  levels.  Since  performance 
standards  incorporate  many  problem-solving  and  high- 
er-level thinking  skills,  our  members  have  also  done  a 
great  deal  of  work  in  the  development  of  performance 
tasks  and  scoring  rubrics  which  are  considered  a more 
authentic  approach  to  measurement  than  the  more  tra- 
ditional multiple-choice  tests.  These  tasks  require  that 
students  produce  a product  rather  than  select  an 
answer  to  a question. 

The  development  of  Benchmarks  by  researchers  and 
educators  in  the  Toronto  Board  of  Education  provides 
an  example  of  an  innovative  approach  to  assessment 
by  an  individual  school  board.  Benchmarks  are  samples 


of  performance  tasks,  along  with  rubrics  and  exemplars,  which 
teachers  can  use  to  make  judgments  about  how  well  their  stu- 
dents are  doing.  Assessment  of  this  kind,  where  the  rubrics  and 
exemplars  are  very  visible  to  parents  and  teachers  as  well  as  to 
students,  is  frequently  referred  to  as  “standards-referenced  assess- 
ment.” The  Toronto  Board’s  Benchmarks  provides  a model  of  good 
assessment  practice.  Teachers  are  encouraged  to  use  benchmarks 
to  develop  similar  tasks  and  to  collect  samples  of  their  students’ 
work  in  portfolios.  When  teachers  conference  with  parents,  the 
samples  of  student  work  in  the  portfolios  are  used  to  talk  about 
progress,  strengths  and  areas  for  growth. 

Public  Reporting 

Efforts  to  improve  the  communication  with  parents  and  the 
larger  community  reflect  the  increased  commitment  of  school 
boards  to  accountability  both  for  their  successes  and  areas  in 
which  program  improvements  are  needed.  For  example,  since 
1994/95,  the  North  York  Board  of  Education  has  annually  pub- 
lished School  Profiles  for  individual  schools  and  the  system  as  a 
whole.  These  profiles  include  the  results  of  system-wide  literacy 
and  mathematics  assessments.  The  reports  are  particularly  useful 
because  they  place  the  results  in  context  by  providing  informa- 
tion about  the  demographic  characteristics  of  the  student  popula- 
tion, as  well  as  information  about  school  programs  and  initiatives. 

Over  the  past  few  years,  the  Scarborough  Board  of  Education 
has  regularly  published  school  and  system  results  from  both 
provincial  and  board-initiated  assessments.  Other  boards  across 
the  province  have  developed,  or  are  in  the  process  of  developing, 
a number  of  innovative  approaches  to  public  reporting.  The 

results  of  the  1997  Provincial  Grade  3 
The  results  Assessment  will  figure  prominently 
m the  profiles  which  many  boards 
of  the  1997  are  planning  to  publish  in  the  com- 
ing year. 

Provincial  Grade  In  addition  to  the  work  done  by 

researchers  within  their  individual 
3 Assessment  boards,  there  have  been  a number 
of  opportunities  in  the  past  few 
will  figure  years  for  school  board  research 
staff  to  collaborate  on  problems 
prominently  in  of  common  interest.  Such  work 

has  been  particularly  important 
the  profiles  in  terms  of  accountability  in 
that  it  reflects  researchers’ 

which  many  commitment  to  provide  their 
boards  with  accurate,  mean- 

boards  are  planning  mgful  data  about  important, 

and  often  controversial 

to  publish  in  issues.  Such  data  are  not 
only  crucial  to  decision 

the  coming  year.  making  within  individual 


boards,  but  are  also  important  from  the  point  of  view  of  public 
policy  and  understanding  the  comparisons  which  are  sometimes 
made  among  boards  and  schools.  Some  of  the  ongoing  collabo- 
rations among  board  researchers  have  arisen  from  Special 
Interest  Group  (SIG)  workshops  organized  by  AERO  for  its 
members.  For  example,  research  staff  from  the  North  York, 
Toronto,  Scarborough,  and  Hamilton  Boards  of  Education  have 
met  on  a regular  basis  to  discuss  issues  related  to  the  calculation 
of  drop-out  rates  and  to  develop  a common  approach  for  their 
calculation  and  comparison.  More  recently,  a SIG  workshop 
on  issues  and  formulas  for  calculating  an  inner-city  index  led 
to  a plan  tor  periodic  meetings  of  interested  researchers 
from  different  boards.  This  work  not  only  has  the  potential 
to  provide  a common,  contextual  measure  to  assist  in  inter- 
preting information  about  schools,  but  also  provides  a means 
of  documenting  the  special  needs  of  specific  communities. 

Building  Capacity 

An  important  activity  in  which  many  AERO  members  are 
increasingly  involved  focuses  on  efforts  to  build  “capacity” 
and  expertise  among  teachers  and  administrators  to  collect 
and  use  data  effectively  for  school  planning  and  decision 
making.  This  may  include  providing  workshops  to  staff  on 
effective  assessment  strategies,  on  how  to  interpret  assessment 
data,  and  on  how  to  examine  assessment  data  in  light  of 
other  information  to  improve  program  delivery.  Workshops 
of  this  nature  have  been  provided  to  teachers,  administra- 
tors, and  supervisory  officers  in  a number  of  boards.  In  the 
fall  of  1996,  research  and  program  staff  of  the  six  boards  in 
the  Assessment  and  Training  Consortium  (Durham,  Halton, 
Lakehead,  North  York,  Scarborough,  and  Waterloo  County) 
offered  a series  of  workshops  on  these  topics  to  boards 
across  the  province.  Research  staff  in  other  boards  have 
produced  newsletters  to  support  teachers’  professional 
development.  One  example  is  Scarborough’s  Research 
Speaks  to  Teachers  which  summarizes  information  from 
the  research  literature. 

Information  Management 

Other  innovations  in  individual  boards  have  involved 
technology  and  the  development  of  software  that  pro- 
vides schools  with  easier  access  to  their  own  data  and 
the  means  to  track  trends  over  time  or  for  particular 
groups  of  students.  For  example,  Dataquest,  developed 
by  the  North  York  Board  of  Education,  allows  sec- 
ondary schools  to  examine  their  student  achievement 
results,  course  enrolment  trends  and  demographic  data. 
In  addition,  schools  receive  their  student  assessment 
results  on  disk  with  a number  of  pre-programmed  and 
custom  report  options.  The  latter  include  a number 
of  simple  search,  sort  and  summarize  features  (e.g., 
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identify  high  or  low  achievers).  The  York  Region  Board  of 
Education  has  developed  a prototype  of  software  which  will 

he  made  available  to  schools  to 
support  school  improvement  plan- 
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many  AERO 
members  are 
increasingly 
involved  focuses 
on  efforts  to 
build  "capacity" 
and  expertise 
among  teachers.., 


ning.  Research  staff  in  the  Durham 


Board  have  played  a key  role 
in  the  development  of  the  board’s 
new,  fully  integrated  information 
management  system;  this  system 
has  powerful  search  and  indexing 
capabilities  which  will  enhance 
information  access  by  board 
staff. 


AERO  as  an  Organization 

To  support  the  professional 
development  needs  of  its 
members  and  provide 
opportunities  for  making 
professional  connections, 
AERO  sponsors  two 
annual  conferences  which 
regularly  attract  not  only 
educational  researchers, 
but  also  educators  in 
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school  board  program  departments,  school  administrators,  and 
supervisory  officers.  Recent  conferences  have  addressed  such 
topics  as:  issues  of  governance  in  education;  the  role  of  educa- 
tional researchers  as  facilitators;  accessing  and  using  databases 
with  demographic  data;  supporting  school  improvement; 
public  reporting  and  school  profiles;  definition,  assessment 
and  reporting  issues  related  to  literacy;  and  approaches  to 
assessment  and  evaluation  in  the  context  of  an  integrated 
curriculum. 

In  addition,  several  small  workshops  are  organized  each 
year  for  members  to  provide  less  formal,  hands-on  profession- 
al development  opportunities.  Recent  sessions  have  focused 
on  the  following  topics:  the  use  of  technology  to  design  and 
scan  customized  surveys;  resource  allocation  indicators  and 
formulas  for  identifying  “inner-city”  schools;  the  experience 
of  a school  board  in  implementing  performance-based 
assessments;  practical  issues  in  preparing  school  profiles;  use 
of  the  Internet;  approaches  to  qualitative  research;  and 
teacher  action  research.  In  some  cases,  participants  in  these 
sessions  have  formed  a Special  Interest  Group  to  permit 
ongoing  information  sharing  and  dialogue.  131 

For  more  information  about  the  Association  and  its  work,  please  contact 
Dr.  Sandra  Sangster  at  the  North  York  Board  of  Education  (395-8136) 
or  by  E-mail  (Sandra_Sangster@nynet.nybe. on. ca). 
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Would  you  please  tell  me,  which  way  I ought 
to  go  from  here?  That  depends  a good  deal 
on  where  you  want  to  go  from  here. 

— Lewis  Carrol 


The  Context 

Teachers  and  educational  administrators 
in  Ontario  are  implementing  a range  of 
curriculum  reforms  in  a period  of  pro- 
found educational  change.  Many  of  these 


initiatives  have  resulted  from  and  been 
conceptualized  by  a discourse  about  how 
best  to  educate  children  for  the  21st 
century.  The  Radwanski  Report  on 
dropouts  (1987),  the  Royal  Commission 
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Student  education  has  become  a highly  political 
and  public  issue.  A performance  conscious 
culture  has  emerged  due  to  a world-wide 
accountability  movement  and  standardsbased 
reform  that  asks  teachers  to  prepare  students 
to  meet  standards  within  a Canadian,  as  well 


as  a world-wide  context. 

Although  accountability  in  education  is  not  a 
new  phenomenon,  it  presents  new  challenges  as 
we  head  toward  the  21st  century.  Teachers  want 
to  be  more  involved  in  the  assessment  of  their 
students;  parents  and  employees  want  to  know 
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on  Learning  in  Ontario  (1994),  and  The 
Road  Ahead  (1997)  identify  many  of  the 
issues  in  education  that  are  part  of  this 
discourse.  The  government’s  agenda  is 
premised  on  beliefs  that  many  students 
are  not  adequately  prepared  for  postsec- 
ondary education  and  work,  that  students 
need  more  basic  language  and  mathematics 
skills,  and  that  more  accountability  is 
required  — both  financially  and  in  terms 
of  a more  demanding  curriculum. 

The  focus  on  “opportunity,  excellence 
and  accountability”  is  made  clear  in  the 
following  action  directives  taken  from 


what  students  are  learning  and  how  well  they 
are  doing;  and  politicians  want  to  know  how 
well  students  are  doing  in  a world-wide  context. 
In  Ontario  this  past  year,  great  efforts  have 
been  made  to  keep  pace  with  the  world-wide 
focus  on  standards-based  reform.  A new 
standards-based  curriculum  promotes  back-to- 
basics  education  and  the  use  of  more  rigorous 
standards  to  provide  public  accountability. 
Although  standards-based  reform  may  seem 
relatively  benign,  it  remains  an  emotionally 
charged  issue  among  the  various  stakeholders. 

A Brief  History  of  Student 
Evaluation,  Standardized 
Testing,  and  Standards 

Student  evaluation  has  traditionally  been  an 


their  publicly  stated  agenda: 

• develop  a core  curriculum  for  elemen- 
tary and  secondary  schools  which  sets 
standards  for  each  grade  level  and 
establish  testing  at  regular  intervals  to 
ensure  attainment  of  those  standards 

• combine  grades  on  report  cards  with 
the  current  descriptive  information 

• increase  the  number  of  instructional 
days  in  a school  year 

• give  teachers,  parents,  business  and 
labour  more  input  into  educational 
decisions  through  a school-based 
management  system 


activity  that  occurred  at  the  end  of  a lesson, 
a unit  or  a school  year  as  an  evaluative  tool 
from  which  decisions  about  students  and 
schools  were  made  (Earl,  1995).  In  the  1930s, 
the  most  popular  form  of  evaluation  was  the 
standardized  test.  This  pencil  and  paper  test 
complemented  a factory  and  mass  production 
model  of  teaching  introduced  by  Frederick 
Taylor  in  the  early  1900s.  His  system  of  scien- 
tific management,  combined  with  the  influence 
of  the  business-industrial  ideology  in  American 
society  and  education,  led  to  the  factory-like 
school,  in  which  learning  was  broken  down 
into  small  bits  of  knowledge  that  were  taught 
and  learned  sequentially  as  students  progressed 
through  an  educational  assembly  line.  With 
their  multiple-choice  formats,  standardized 


• reduce  secondary  education  from  five 

to  four  years 

The  goal  is  a curriculum  that  features 
“the  basics,”  regular  testing  of  students 
(locally,  provincially,  nationally,  and  inter- 
nationally), and  a province- wide  standard 
report  card  that  features  marks.  The 
constructivist  orientation  and  ambiguity 
associated  with  the  previous  governments 
Common  Curriculum  have  been  replaced 
with  clearer,  more  rigorous,  and  grade- 
by-grade  specific  knowledge  and  skills. 
The  new  Ontario  Curriculum  is  more 
rigorous  in  that  the  required  curriculum 


tests  became  the  perfect  tool  for  measuring 
student  knowledge  and  were  particularly  popu- 
lar because  of  their  cost  effectiveness,  and 
their  capacity  to  test  many  students  at  once 
with  uniformity  and  consistency  (Callahan, 
1962). 

By  the  1960s,  however,  testing  began  to 
reveal  fluctuations  in  student  achievement. 
Educators  and  the  public  became  alarmed 
by  large  expenditures  being  made  toward 
improving  education  when  test  scores  were 
falling  and  when  employers  complained  that 
high  school  graduates  couldn't  read  or  write. 
As  this  was  a particularly  economically 
wealthy  decade,  politicians  had  good  reason  to 
experiment  with  innovations  in  an  attempt 
to  solve  the  problem  of  an  alleged  substandard 


expectations  are  both  more  difficult  and 
more  specific  than  those  of  the  previous 
provincial  curricula. 

Educational  reform  in  Ontario  reflects 
a struggle  world-wide  between  two  very 
different  approaches.  Linda  Darlmg- 
Hammond  (1992)  describes  these  two 
different  views  in  the  following  passage: 

“Two  very  different  theories  of  school 
reform  are  working  in  parallel  and  some- 
times at  cross-purposes....  One  theory 
focuses  on  tightening  the  controls:  more 
courses,  more  tests,  more  directive  cur- 
riculum, more  standards  enforced  by  more 
rewards,  and  more  sanctions....  A second 
theory  attends  more  to  the  qualifications 
and  capacities  of  teachers  and  to  develop- 
ing schools  through  changes  in  teacher 
education,  licensing,  and  certification 
processes... professional  development 
schools,  efforts  to  decentralize  school 
decision  making  while  infusing  knowl- 
edge, changing  local  assessment  practices, 
and  developing  networks  among  teachers 
and  schools.”  (p.22) 

Changes  in  Ontario  education  initially 
focus  on  the  first  of  these  theories  of 
school  reform;  subsequent  changes  will 


education. 

The  growing  public  concern  for  account- 
ability resulted  in  more  standardized  testing.  It 
can  safely  be  said  that  in  the  U.S.  at  least,  the 
major  precedent  for  testing  and  standard 
setting  was  the  political  reaction  to  critics 
of  education.  To  illustrate  the  magnitude  of 
standards-based  reform  in  the  U.S.  in  the  past 
decade,  state-wide  assessment  programs  have 
grown  from  29  states  in  1982  to  46  in  1992. 
In  addition,  American  professional  associa- 
tions like  the  National  Council  of  Teachers  of 
Mathematics  (NCTM),  the  National  Council  of 
Teachers  of  English  (NCTE),  and  the  National 
Council  for  Social  Studies  (NCSS)  began  to 
develop  educational  standards  in  their  own 
disciplines.  Although  the  setting  of  standards 
has  been  a more  recent  phenomenon  in 
Ontario,  the  movement  toward  standards-based 
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deal  with  building  capacity  of  teachers 
and  developing  schools.  The  creation  of 
centralized  standards,  frameworks  for 
increased  accountability,  decentralized 
decision-making,  centrally  allocated  finan- 
cial resources,  and  the  acceptance  of 
principles  founded  on  a market  based 
economy  characterize  education  reform 
currently  underway  in  many  of  the  devel- 
oped countries.  Ontario  is  no  exception. 

Research  on  Standards 

Establishing  curriculum  standards  is  not  a 
simple  matter  (see,  for  example,  Portelli  & 
Vibert,  1997;  Resnick  & Nolan,  1995). 
Definition  is  an  initial  hurdle.  Polemically 
“standards”  is  used  in  many  different  ways. 
In  the  educational  literature,  the  meaning 
and  use  of  content  standards,  performance 
standards,  and  program  standards  are  hotly 
debated.  For  some,  promoting  the  notion 
of  standards  is  seen  as  part  of  an  elitist 
approach  to  education.  For  others,  the 
focus  on  curriculum  content  rather  than 
performance  is  limiting.  And,  for  others, 
the  use  of  standards  is  a way  of  supporting 
quality  learning  and  raising  achievement 
for  all  students. 


While  focussing  on  standards  (however 
defined)  may  have  the  potential  to  raise 
achievement  for  all  students,  it  will  take 
hard  work  and  time.  Several  key  issues 
often  associated  with  the  use  of  standards- 
based  curriculum  reform  that  are 
frequently  reported  in  the  literature 
include  the  following: 

• standards  that  are  set  must  be  credible 
and  relevant 

• the  number  of  standards  must  be 
manageable 

• terminology  must  be  simplified  and 
clarified 

• implementation  issues  must  always  be 
considered 

• the  alignment  of  instruction  and 
assessment  with  standards  is  critical 

• standards  should  be  coherent  with  cur- 
rent views  of  learning  and  instruction 

• a phase-in  for  standards  and  appropri- 
ate assessments  is  necessary 

• the  defined  curriculum,  the  taught 
curriculum,  the  learned  curriculum, 
and  the  assessed  curriculum  must  be 
understood  and  aligned 

• how  good  is  good  enough  should 
be  continually  probed  and  exemplars 


reform  in  education  has  been  no  less  rigorous 
or  political. 

The  Canadian  Context 

According  to  Nagy  (1995),  two  significant 
events  acted  as  the  catalysts  for  Ontario's 
subscription  to  standard  setting:  first,  the 
end  of  provincial  graduation  exams  in  many 
provinces  reduced  levels  of  accountability; 
second,  an  increase  in  an  immigrant  popula- 
tion between  1965  and  1992,  resulted  in 
perceived  lower  literacy  rates.  Consequently, 
a renewed  interest  in  standards  and  account- 
ability procedures  has  emerged  this  past 
decade  in  Canada.  The  political  agenda  has 
been  set  to  provide  the  public  with  the  infor- 
mation necessary  to  assess  the  effectiveness  of 
schools  and  to  reflect  world-class  standards. 


Ontario  is  not,  however,  exempt  from  the 
problems  of  standards-based  reform,  although 
our  latent  adoption  of  this  reform  has  given 
Canadians  the  luxury  of  learning  from  others' 
mistakes.  Ontario  is  still  vulnerable  to  charges 
made  by  critics  that  standardized  tests  measure 
the  wrong  skills;  distort  classroom  instruction; 
falsely  reassure  parents;  and  discriminate 
against  minority  and  special  needs  students. 
Ontario  has,  however,  been  able  to  avoid  some 
of  the  pitfalls  experienced  by  other  countries. 
For  example,  testing  in  the  U.S.  has  in  the  past 
primarily  been  commercially  controlled  by 
external  agencies  and  publishers  that  develop 
the  standardized  tests  with  no  or  little  teacher 
input  (Darling-Hammond,  1994).  (The  U.S.  has 
since  re-examined  their  assessment  programs 
and  has  moved  toward  other  testing  procedures 
besides  commercially  available  tests.)  The 


u 

The  creation  of  centralized  standards,  frameworks  for  increased 
accountability,  decentralized  decision  making,  centrally 
allocated  financial  resources,  and  principles  founded  on  a market 
based  economy  characterize  education  reform  in  many  of  the 
developed  countries. 
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of  student  performance  must  be 
continuously  collected  and  examined 

• re-educating  teachers  to  work  in  new 
ways  is  required 

• vigorous  discussions  and  conversations 
about  curriculum  frameworks,  with  a 
focus  on  student  learning,  are  required 

The  Provincial  Standard 

The  recent  development  of  new  Ontario 
Curriculum  documents  in  Language  and 
Mathematics,  Grades  1-8,  involved  conver- 
sations among  a range  of  stakeholder 
groups.  Curriculum  policy  decisions 
emphasized  clear,  rigorous,  grade-by-grade 
direction  as  to  what  is  to  be  accomplished 
and  how  achievement  will  be  measured. 
The  means  of  accomplishing  the  task  — 
how  to  do  it  — however,  is  not  mandated 


by  policy;  the  method  of  implementation 
is  the  responsibility  of  schools,  teachers,  par- 
ents, and  local  community.  The  curriculum 
expectations  are  also  written  so  that  parents, 
not  just  teachers,  can  understand  them. 

The  Ontario  Curriculum  uses  a four- 
level  achievement  scale  for  determining 
“how  well”  students  demonstrate  what 
they  are  taught.  These  publicly  available 
categories  and  descriptions  of  knowledge 
and  skill  achievement  at  four  different 
levels  are  intended  to  be  applied  to  a 
student’s  work  for  any  given  grade. 
Descriptions  of  the  four  levels  of  achieve- 
ment provide  teachers  with  information 
that  can  anchor  student  assessments;  the 
levels  provide  a way  for  teachers  to  judge 
the  quality  of  a student’s  work. 

Level  3 is  the  provincial  standard. 


Teachers  and  parents  can  be  confident  that 
students  who  achieve  at  Level  3 are  well- 
prepared  for  work  at  the  next  grade;  Level 
4 achievement  surpasses  the  provincial 
standard.  Level  1 identifies  achievement 
that  falls  much  below  the  provincial  stan- 
dard specified  for  the  grade;  Level  2 iden- 
tifies achievement  that  approaches,  but  is 
not  yet  at,  the  provincial  standard  specified 
for  the  grade. 

The  EQAO  addresses  the  accountabili- 
ty focus  through  regular  province-wide 
assessments.  In  their  recent  Grade  3 assess- 
ment program,  for  example,  every  effort 
was  made  to  gather  information  about  the 
achievement  of  students  in  a real-world 
context.  Because  the  assessments  are  close- 
ly aligned  with  existing  curriculum  policy, 
teaching  to  the  test  means  teaching  the 
curriculum.  In  fact,  the  ten-day  unit  was 
designed  as  a learning  unit  that  contained 
test  items  and  assessment  tasks. 

The  provincial  report  card  is  also  con- 
nected directly  to  the  specified  curriculum 
expectations  and  the  levels  of  achievement 
for  the  grade.  It  ensures  that  all  teachers  in 
Ontario  use  a standard  procedure  for 
reporting  student  progress.  The  report  card 


newly  created  Education  Quality  and 
Accountability  Office  in  Ontario,  an  arms-length 
agency  funded  by  the  provincial  government, 
employs  alternative  test  development  and 
administration  procedures,  to  avoid  removing 
teachers  from  the  assessment  process.  Teacher 
involvement  in  Ontario  is  most  recently  evident 
in  both  the  creation  and  marking  of  the  Grade 
3 tests  administered  this  past  spring  of  1997. 

A sampling  of  debates  still  to  be  resolved 
in  Ontario  include  whether  to  test  every  student 
(census  testing)  when  administering  large-scale 
tests,  or  to  test  only  a representative  sampling 
of  students.  The  decision  of  what  should  be 
tested  remains  a nagging  question,  and  is 
often  decided  by  arbitrarily  including  what 
is  easy  to  assess.  Nagy  (1995)  warns  that 
provincial  testing  in  Ontario  focuses  on  only  a 
few  subjects,  hence  lowering  the  value  of  other 


subjects.  The  new  Ontario  Curriculum  does 
provide  directives  to  gain  consistency  between 
grades,  although  borderline  grade  performances 
still  require  that  educators  rely  on  their  judg- 
ment especially  in  decisions  at  the  border 
between  one  grade  and  the  next. 

A consequence  of  past  testing  in  the  U.S. 
has  resulted  in  charges  of  a decline  in  higher- 
order  skills,  as  more  and  more  emphasis 
is  placed  on  rote  learning  of  basic  skills. 
Standardized  testing  should  include  varying 
assessment  approaches,  a task  that  EQAO  has 
already  attempted  to  account  for  by  including 
holistic  and  performance  assessment  compo- 
nents in  their  large-scale  tests;  however,  assess- 
ment which  becomes  more  authentic  also 
becomes  more  difficult  to  standardize.  Canadian 
teachers  still  face  having  to  match  classroom 
assessment  strategies  with  standardized  testing. 


This  is  a particularly  difficult  task  since  stan- 
dardized tests  remain  external  to  the  teaching 
and  learning  that  goes  on  in  the  classroom. 
Teachers  also  find  that  their  professional  judg- 
ment and  their  ability  to  provide  good  instruction 
is  curtailed  by  the  need  to  teach  students  to 
succeed  on  standardized  tests  (Stiggins,  1994). 

Conclusion 

For  siandards-based  reform  to  succeed,  provin- 
cial, national,  and  international  shared-meaning 
of  the  purposes  and  definitions  of  standards 
is  necessary.  Still  more  research  is  needed  to 
gain  a better  understanding  of  international 
experiences  with  standard-based  reform.  Less 
attention  should  be  given  to  the  ranking  of 
countries,  and  more  to  knowing  the  weaknesses 
and  strengths  of  each  system  in  order  to  learn 
from  them.  Perhaps  the  best  place  to  begin 


Teachers  need  to  have  an  intimate  understanding  of  the 
curriculum  expectations  and  the  standards  as  operationalized 
by  the  achievement  level  descriptions. 


is  available  electronically  and  is  designed  to 
provide  clear,  precise  information  to  par- 
ents about  the  achievement  of  all  students. 

On  the  surface,  curriculum  reform 
is  simple.  Making  it  work,  however,  is 
complex.  Teachers  need  to  have  an  inti- 
mate understanding  of  the  curriculum 
expectations  and  the  provincial  standard 
as  operationalized  by  the  achievement 
level  descriptions.  Staff  development  and, 
in  particular,  accountability  and  assessment 
leadership  are  required  so  that  wise  eyes 
prevail. 

Samples  of  students’  work  for  each 
achievement  level  are  practical  ways  for 
communicating  progress.  Once  portfolios 
containing  samples  of  students’  work 
become  available,  teachers  can  use  them 
to  make  rounded  judgments  that  are  based 
on:  the  knowledge  of  how  the  student 
performs  in  various  contexts,  the  strengths 
and  weaknesses  of  the  student’s  perfor- 
mance, and  the  selection  of  the  best 
achievement  level  to  portray  the  student’s 
performance. 

Conclusion 

Curriculum  reform  is  not  sell-implement- 


ing. The  first  steps  have  been  taken  in 
the  short  life  span  of  curriculum  renewal 
in  Ontario  education.  The  terrain  is  rela- 
tively new  and  unknown.  The  intent  in 
Ontario  is  to  strengthen  the  curriculum, 
promote  better  learning,  raise  student 
achievement  and  increase  accountability. 
Agreement  about  what  constitutes  appro- 
priate curriculum  and  achievement  often 
depends  on  purpose,  time,  place,  and 
values.  In  Ontario  the  use  of  a provincial 
standard  in  curriculum  is  in  its  infancy. 
Undoubtedly  the  conversations  among 
all  stakeholders  will  continue.  Such 
conversations  will  provide  a forum  for 
accountability  that  in  turn  will  shape 
educational  practice.  Curriculum  reform 
has  arrived  in  Ontario.  What  difference 
it  will  make  for  students’  learning 
and  achievement  will  become  clear 
over  time.  ra 
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educating  stakeholders  about  standards-based 
reform  is  in  preservice  training,  professional 
development,  and  education  campaigns  to 
explain  new  assessment  methods,  standards 
and  standardized  testing  to  the  public,  teach- 
ers, students  and  employers.  More  importantly, 
the  use  of  standards  must  make  direct  use 
of  teachers'  judgments  and  include  a process  in 
which  students  can  be  involved. 

There  are,  and  will  always  be,  sceptics  who 
dismiss  standards-based  reform  as  a passing 
fad,  saying  that  the  problems  are  too  great  to 
arrive  at  shared  meaning  and  the  resources, 
too  little.  These  critics  would  like  to  see  money 
allocated  for  programs  that  have  proven  suc- 
cessful and  that  can  guarantee  high  student 
achievement.  Others,  less  sceptical,  yet 
cautious,  frown  on  mandated  standards  with 
minimum  competency  levels  and  a national 


curriculum  as  its  framework  (Eisner,  1991). 
Instead,  they  claim  the  use  of  standards  should 
be  voluntary,  all  the  while  promoting  high 
student  achievement  that  is  realistic  for  the 
individual  school  and  school  district.  Standards 
need  to  address  not  only  the  cognitive  aspect 
of  learning,  but  also  need  to  take  into  consider- 
ation the  student's  social  growth.  Perhaps  the 
greatest  challenge  of  standards-based  reform 
will  be  a revision  of  budgetary  policies  to  ensure 
equitable  outcomes. 

The  varying  opinions  and  differing  under- 
standing of  standards  and  testing  among  stake- 
holders indicate  that  standards-based  education 
presents  new  challenges  in  Ontario,  as  we  head 
toward  the  21st  century.  Whether  pejorative, 
political  or  visionary  in  its  reform  appeal,  an 
ongoing  dialogue  is  needed  as  educators  struggle 
to  make  sense  of  standards-based  reform.  D 
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education,  some  of  the  confusion  about  standards  comes 
from  the  range  of  meanings  that  get  assigned  to  the 
word.  In  the  fall  of  1996,  we  attempted  to  clarify  the  notion  of 
standards  in  Ontario  education  by  asking  three  academics  (Bob 
Wilson,  Joel  Weiss,  and  Graham  Orpwood)  to  present  their  ideas 
to  staff  from  the  Ministry  of  Education  and  Training,  OISE/UT, 
the  Education  Quality  and  Accountability  Office,  and  the 
Association  of  Educational  Researchers  of  Ontario.  This  article 
is  an  edited  summary  of  their  remarks. 

Bob  Wilson,  Queen’s  University 

At  least  eight  of  the  definitions  for  stan- 
dards in  the  Oxford  English  Dictionary  have 
some  potential  for  describing  our  use  of 
the  term  in  Ontario  education.  The  first 
says:  “the  standard  of  which  others  are 
copies,  and  to  which  the  ultimate  appeal 
must  be  made.”  To  some  extent,  this 
definition  holds  when  the  Ministry  provides  copies  of  “standards” 
to  be  followed  in,  say,  mathematics. 

Another  definition  states:  “the  legal  magnitude  of  a unit  of 
measure  or  weight.”  This  definition  holds  when  the  Ministry 
provides  rubrics  for  key  outcomes  that  describe  differing  levels  of 
achievement  possible  according  to  its  standards. 

When  achievement  is  categorized  according  to  certain 


qualitative  judgments  (Level  1 and  2 are  not  quite  there.  Level  3 
is  good.  Level  4 is  “beyond”),  it  suggests  that  there  are  common 
units  of  measurement  being  applied  across  outcomes,  units  that 
have  a uniform  criterion.  These  two  aspects,  uniformity  of  mea- 
sures and  common  units,  constitute  two  more  of  the  Oxford  s 
definitions.  If  all  subject  matter  were  categorized  using  the  same 
levels,  then  a further  degree  of  standardization  would  occur. 

Another  definition  provides  an  indication  of  the  context  in 
which  the  standard  is  to  be  applied:  “the  substance  or  thing 
which  is  chosen  to  afford  the  unit  measure....”  Assignments 
and  tests  made  by  teachers,  and  measures  developed  under  the 
auspices  of  EQAO,  are  both  using  this  definition  when  they 
are  referenced  to  the  standards  themselves. 

Still  another  definition  refers  to  a standard  as  “a  rule,  principle, 
or  means  of  judgment  or  estimation;  a criterion  measure.” 
Standards  are  expected  to  serve  as  the  ruler  against  which  judg- 
ments of  curricular  relevance,  assessment  appropriateness,  instruc- 
tional congruence,  and  resource  usefulness  can  be  assessed. 

Notice  that  many  of  these  aspects  of  the  usefulness  of  standards 
may  not  apply  solely  or  even  mainly  to  classrooms  and  schools, 
but  to  the  resources  provided  to  educators  to  do  their  work. 

That  standards  should  represent  worthwhile  attainments  for 
children  is  indicated  in  the  last  definition:  “a  definite  level  of 
excellence,  attainment,  wealth,  or  the  like,  or  a definite  degree  of 
any  quality,  viewed  as  a prescribed  object  of  endeavor....”  This 


u the  standard  definition  leads  to  a discussion  of 

how  any  particular  standards  are 
Of  Which  Others  arrived  at,  who  participated  in  their 
development,  and  do  they  indeed 

are  copies,  and  represent  what  Ontarians  desire 

for  their  children. 

to  which  the  The  major  role  in  the  devel- 
opment of  any  set  of  standards 
ultimate  appeal  must  be  given  to  those  who  are 

expert  and  experienced  in  the 
mUSt  be  made.  field.  Only  these  people  will 

have  the  technical  and  con- 
ceptual expertise  to  decide  whether  a given  standard  is  desirable  and 
attainable.  These  two  judgments  (desirability  and  attainability)  com- 
bine what  we  in  education  have  traditionally  differentiated  into 
“norm-referenced”  and  “criterion-referenced”  assessments.  Standards 
not  only  provide  the  criterion  or  standard  for  which  all  should 
strive,  but  they  also  describe  the  variability  expected  when  groups 
of  individuals  with  different  degrees  of  ability  attempt  a task. 

Important  though  the  influence  of  professional  judgments  is  in 
the  development  of  standards,  they  are  by  no  means  sufficient. 
Indeed,  if  the  standards  are  to  be  used  as  “level  of  excellence, 
attainment,  wealth,  or  the  like,”  they  must  be  seen  to  satisfy  the 
definitions  of  quality  in  educational  attainments  by  many  relevant 
parties.  There  may  not  be  much  dissent  about  the  statements 
themselves,  but  there  is  a high  likelihood  that  there  will  be  dissent 
about  levels  of  proficiency  (“expected”  and  “beyond  expecta- 
tions” for  example)  if  standards  are  determined  only  by  educators. 

Unless  there  is  some  general  consensus  within  the  community 
about  the  degrees  to  which  we  expect  students  to  meet  certain 
standards  — and  what  follows  from  that,  the  acceptability  of 
results  that  show  how  many  are  actually  meeting  them  — the 
standards  will  become  a source  of  discord  instead  of  becoming 
the  “authoritative  or  recognized  exemplars”  that  are  above  the 
interpretive  fray. 

Something  like  this  point  of  view  no  doubt  gave  impetus  to 
the  creation  of  an  “arms-length”  agency,  the  Education  Quality 
and  Accountability  Office  (EQAO),  to  perform  accountability 
exercises  for  the  province.  The  independence  of  EQAO  from  the 
Ministry  of  Education  and  Training  indicated  the  desire  to  have  a 
disinterested  judgment  about  the  attainments  of  Ontario’s  students. 
EQAO  must  work  from  the  standards  as  a basis  for  collecting  and 
reporting  achievement  data  for  Ontario  students,  but  it  will  need 
to  avoid  evaluative  heading  or  labels  and  encourage  some  sort  of 
public  interpretation  process  to  give  their  audits  credibility. 

Interpretive  decision-making  concerning  attainments  is  espe- 
cially necessary  because  of  the  potential  for  inappropriate  equating 
of  indicators  with  standards.  While  rulers  exist  in  many  scientific 
and  technical  fields  that  do  not  change  from  one  instance  to 
another,  this  does  not  apply  in  education.  In  education,  the  entities 
being  measured  — students  and  to  some  extent  their  teachers  — 


can  adapt  to  the  tool  being  applied  to  them,  and  no  doubt,  will. 

Using  the  same  ruler  over  and  over  again  in  education  pro- 
vides an  incentive  to  practise  on  the  characteristics  of  the  ruler 
and  not  on  the  substance  of  what  the  ruler  is  measuring.  The 
drilling  of  students  on  sample  items  on  standardized  tests,  college- 
preparatory  classes  for  taking  entrance  examinations,  and  the  habit 
of  stopping  the  curriculum  half  way  through  the  term  to  prepare 
tor  end  of  school  government  examinations  are  all  examples  of 
this  tendency.  EQAO’s  praiseworthy  attempt  to  mimic  as  much  as 
possible  “good  instructional  practice”  in  its  assessment  should 
mitigate  this  impulse,  and  perhaps  even  turn  it  to  advantage. 

It  is  a commonplace  in  non-academic  discussions  to  confuse 
standards  with  test  results.  That  is  why  interpretation  of  results  is 
necessary,  and  why  the  interpretation  process  must  focus  on  the 
quality  of  the  achievements  shown  on  this  particular  “substance 
or  thing... chosen  to  afford  the  unit  measure”  and  not  on  some 
arbitrary  judgment  about  quality  made  in  advance.  Done  proper- 
ly, an  interpretation  process  can,  of  itself,  add  to  the  acceptance  of 
the  results  as  evidence  of  the  degree  of  attainment  of  the  stan- 
dards. Done  improperly,  or  not  at  all,  it  will  only  lead  to  rejec- 
tions ot  the  process  by  key  members  of  the  society,  especially 
those  ot  a particular  ideological  bent  whose  favored  judgments, 
made  in  advance,  are  not  being  supported  by  the  data. 

I There  is  no  magic  wand  that  can  be  waved 

■K i5|  Ordinary  language  usage  connotes  “rigour” 

upon  features  and  criteria  that  just  require 
S*-'*-''"  appropriate  development  techniques.  This 

is  far  from  the  case  when  values  and  judgments  about  values 
come  into  play. 

In  pluralistic  societies,  such  as  Canada  and  the  U.S.,  education 
systems  are  forged  in  communities  with  different  interests,  views, 
and  underlying  values.  Considerations  about  curriculum  often  are 
subject  to  a pendulum  effect  where  changes  at  least  in  the 
rhetoric  of  “what”  should  be  taught,  for  “which”  purposes, 
and  “how”  it  is  to  be  learned  occur  over  time  (e.g.,  from  child- 
centered  to  technology-driven  views). Very  often,  different 
orientations  to  curriculum  represent  diverse  emphases  on  what 
should  be  taught,  for  whom,  how  and  with  what  resources. 

My  view  is  that  consensus  about  standards  is  dependent  upon 
agreement  on  the  curriculum.  Since  this  is  difficult  to  achieve, 
expectations  for  full  agreement  on  standards  are  unlikely.  Any  dis- 
cussion about  curriculum  requires  an  understanding  that  curricu- 
lum has  different  faces.  For  some,  curriculum  refers  to  a syllabus,  a 
course  of  study  which  is  derived  from  the  Greek  currere,  to  run 
the  track  or  course.  For  others,  it  is  synonymous  with  subject 
matter  content,  what  some  consider  knowledge.  Yet  another  use 
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of  the  term  addresses  the  resources  and  processes  needed  to  bring 
about  the  desired  outcomes.  The  experiential  impact  of  being 
exposed  to  curriculum  — that  is,  the  experienced  curriculum,  is 
yet  another  face. 

A popular  strategy  with  researchers  conducting  provincial  and 
international  assessments  is  to  compare  “intended,”  “actual,”  and 

“attained”  curricula.  Layering  all 
"Without  references  to  “curriculum”  is  a con- 
cern with  jurisdiction.  Curriculum 
inclusion  of  discussions  and  practices  may  relate 
to  everything  from  the  fairly 
program  global  comparisons  of  curriculum 
across  different  countries  to  the 
standards,  this  individual  curricular  moments 
between  student  and  teacher. 

represents  a There  are  different  levels  of 

curricular  policymaking, 

version  of  an  input-  development,  implementation, 

and  assessment  (federal, 

output  model  and  provincial,  board  or  district, 

school,  classroom,  teacher 

suggests  the  metaphor  federations,  parent  coun- 
cils) . At  each  level,  a range 
of  a 'black  box7,..."  of  participants  from  poli- 
cymakers to  educators  to 

students  to  parents  to  community  members  are  involved  in  differ- 
ent roles.  With  the  complexities  of  roles,  levels,  and  previous 
experiences  it  is  not  surprising  that  consensus  is  rarely  achieved. 

Just  as  there  are  different  faces  of  curriculum,  I can  refer  to 
different  “facets”  of  standards: 

Content  standards  refer  to  the  questions  of  what  students 
should  know  and  be  able  to  do  — that  is,  knowledge  and  skills. 

Program  standards  address  the  “how”  question:  descriptions  of 
the  resources  available  and  necessary  and  the  activities  appropriate 
for  students  to  learn  the  content  standards.  These  include  instruc- 
tional and  curricular  designs,  techniques,  course  structures, 
recommended  activities  of  what  should  happen  in  the  learning 
setting,  including  available  resources  and  modes  of  presentation. 

Performance  standards  refer  to  the  assessment  procedures  for 
determining  the  attainment  of  the  content  standards.  These  are 
the  indicators  of  success  that  we  develop  and  use  in  the  form 
of  provincial  audits,  comparative  surveys  at  the  national  (SAIP) 
and  international  levels  (IEA),  board  procedures  (Benchmarks), 
and  the  like. 

Taken  together,  content  and  performance  standards  are  a 
reflection  of  a popular  approach,  the  “curriculum-as-objectives” 
or  outcomes-based  movement.  Most  of  the  literature  on  standards 
refers  to  either  content  or  performance,  or  a combination  of  the 
two,  in  the  form  of  specification  of  objectives,  and  the  techniques 
and  procedures  for  assessment.  Without  inclusion  of  program 
standards,  this  represents  a version  of  an  input-output  model  and 


suggests  the  metaphor  of  a “black  box,”  in  which  there  is  the  like- 
lihood of  not  knowing  what,  if  anything,  would  bring  about  the 
results. 

Program  standards  are  necessary  ingredients  in  moving  beyond 
the  black  box  conundrum.  First,  specification  of  curricular  and 
instructional  features  puts  flesh  on  the  “intended  curriculum”  and 
moves  it  beyond  a consideration  of  content.  Second,  it  provides 
more  leverage  in  determining  whether  the  intentions  are  being 
met  in  practice.  The  introduction  of  program  standards  allows  for 
a more  holistic  appreciation  of  the  concept  of  curriculum  stan- 
dards for  it  allows  us  to  make  judgments  about  the  fit  between 
program  standards  and  the  actual  practices. 

Making  the  case  for  program  standards  does  not  necessarily 
suggest  that  such  standards  are  easily  developed.  This  process  is 
fraught  with  all  the  complexities  inherent  in  the  multi-faceted 
views  on  curriculum.  I suspect  the  reason  why  we  pay  so  little 
attention  to  program  standards  is  that  it  is  difficult  to  gam  con- 
sensus and  develop  technically  proficient  procedures.  Overarching 
both  of  these  considerations  is  my  view  that  it  is  difficult  to 
employ  universal  standards  in  different  curricular  situations.  How 
can  we  be  certain  that  the  same  program  standards  apply  to 
different  locales,  with  different  resources,  student  and  teacher 
experiences?  We  can  have  some  relatively  simple  measures,  such 
as  determination  of  whether  students  have  the  “opportunity-to- 
learn”  the  curriculum.  Even  assuming  the  ability  to  develop  such 
a measure,  anything  more  complex  will  be  difficult  to  develop  or 
to  implement  consistently. 

I return  to  my  original  point  about  standards  being  dependent 
upon  gaining  consensus.  The  development  and  use  of  standards  is 
influenced  by  at  least  three  considerations  that  involve  choices 
and  judgments:  educational  (what  should  be  done),  technical 
(what  can  be  done),  and  political  (what  will  be  done). 
Unfortunately,  the  technical  and  political  considerations  often 
dominate  standard-setting  practices  because  the  educational  con- 
sideration is  most  at  play  in  developing  program  standards;  it 
should  come  as  no  surprise  that  it  has  become  the  “step-child”  in 
the  standards  literature. 

Graham  Orpwood,  York  University 

Standards  are  of  little  use  unless  they  are 
implemented  by  school  boards  and  teachers 
as  the  basis  for  developing  curriculum  and 
classroom  assessment.  And,  if  they  are  to  be 
used  to  influence  practice,  they  must  first 
command  respect.  I would  like  to  address 
the  issue  of  respect  and  its  counterpart 
concept,  “authority.”  The  two  often,  but  not  always,  go  together. 
One  may  or  may  not  “respect”  authority  and  this  depends  on  the 
basis  for  that  authority.  The  issue  is  further  complicated  because 
of  the  different  ways  in  which  we  use  the  word  “authority” 
and,  to  clarify  these,  let  me  turn  to  the  work  of  Richard  Peters, 


"Great 
teachers  need 
not  resort  to 
the  formal 
authority  vested 


a distinguished  contributor  to  the 
field  of  philosophical  analysis  applied 
to  educational  concepts. 

Peters  (1959)  distinguishes 
between  two  essential  concepts 
ot  authority,  as  denoted  by  the 
phrases  “in  authority”  and  “an 
authority.”  We  say  that  a person 
is  “in  authority”  if  (s)he  is  in 
a position  which  is  endowed 
in  their  position/7  with  the  legal  or  administra- 
tive right  to  command  the 

obedience  of  those  “under”  that  authority.  A general  is  in  author- 
ity over  the  troops  in  the  army,  the  Pope  is  in  authority  over  the 
members  of  the  Roman  Catholic  Church,  and  the  Premier  of 
Ontario  over  the  members  ot  his  government.  In  an  educational 
context,  a school  principal  is  similarly  in  authority  over  a school 
and  a teacher  over  a class.  This  authority  is  conferred  on  the  indi- 
vidual by  virtue  of  the  position  which  (s)he  occupies  rather  than 
by  the  individuals  personal  knowledge,  skills  or  virtues  (though 
these  may  or  may  not  have  been  the  basis  for  their  appointment 
in  the  first  place). 

However,  we  also  use  the  word  “authority”  to  denote  an  indi- 
vidual (or  an  institution  or  even  a book)  that  has  such  a level 
of  knowledge  or  skill  in  a given  field  that  he  or  she  (or  it)  is 
widely  respected  in  that  field.  In  this  sense,  some  lawyers  can  be 
described  as  “constitutional  authorities”  or  scientists  as  “authori- 
ties” in  their  specific  branch  of  science,  and  so  on  (even  the 
Oxford  English  Dictionary  is  said  to  be  an  authority  in  this  sense). 
Such  authority  is  conferred  not  by  formal  democratic  or  legal 
process  but  through  recognition  by  a significant  body  of  people 
of  the  value  of  the  individual’s  expertise  acquired  through  study 
or  experience.  When  the  person  ceases  to  have  a formal  role  in 
society  this  form  of  authority  does  not  disappear  as  it  is  part  of 
the  person,  not  externally  conferred  for  a limited  period  of  time. 

Ideally  the  two  go  together  and  great  teachers  need  not  resort 
to  the  formal  authority  vested  in  their  position  because  their 
words  are  respected  by  those  whom  they  teach  (this  was  Peters’ 
original  point);  however,  it  can  also  be  otherwise.  A person  can  be 
in  authority  and  totally  lacking  the  respect  of  those  below  him  or 
her.  Similarly  one  can  be  recognized  and  respected  as  an  authori- 
ty in  some  area,  yet  have  no  mandate  to  influence  others. 

How  does  all  this  apply  to  the  development  of  standards? 
Clearly  the  Ministry  of  Education  and  Training  (MET)  is  “in 
authority”  over  curriculum  in  the  province.  Indeed,  successive 
Ministers  have  taken  pains  to  claim  and  assert  that  authority  since 
taking  office.  But  who  are  the  authorities  (in  the  other  sense)  in 
teaching  and  learning,  in  curriculum  planning  and  assessment? 
Surely,  as  the  Ministry  shrinks  smaller,  they  cannot  claim  to  being 
the  exclusive  authorities  in  how  best  to  develop  curriculum.  I 
would  assert  that  there  are  educators  throughout  the  province 


who  have  demonstrated  their  expertise  and  skill  sufficiently  over 
many  years  to  be  recognized  as  authorities  in  this  field.  However, 
they  have  no  mandate  to  influence  provincial  curriculum. Yet, 
unless  these  two  forms  of  authority  can  be  reconciled,  it  is  possi- 
ble that  any  standards  developed  will  not  be  implemented  or  have 
the  desired  impact. 

There  are  many  examples  of  curriculum  being  developed  that 
have  authority  in  the  latter  sense,  in  that  they  are  the  product  of 
classroom-based  experience  and  up-to-date  research  but  which 
have  never  received  the  “imprimatur”  of  official  MET  sanction. 
Similarly,  there  are  policy  documents  of  the  MET  that  have 
received  all  the  authority  of  ministerial  approval  but  which  have 
never  commanded  the  respect  of  the  majority  of  educators  in  the 
field.  The  converse  is  also  true,  of  course.  But  therein  lies  the 
dilemma  of  standards  development.  How  can  a standards  docu- 
ment represent  both  forms  of  authority?  How  can  it  achieve  both 
respect  and  official  status? 

The  Assessment  of  Science  & Technology  Achievement  Project 
(ASAP)  had  this  dilemma  in  mind  when  it  was  planned.  ASAP 
was  designed  as  a collaborative  project  of  York  University  Faculty 
of  Education  and  17  school  boards  in  Southern  Ontario.  The 
ASAP  standards  were  developed  by  a process  involving  300  class- 
room teachers  working  with  a project  coordinator  based  at  York 
University.  In  addition,  the  project  drew  from  detailed  analyses  of 
science  and  technology  standards  documents  from  the  ASAP 
boards,  from  the  rest  of  Canada,  and  from  international  jurisdic- 
tions. The  draft  standards  were  developed  through  a series  of  40 
workshops  over  the  1995/96  school  year  followed  by  field  trials 
m schools  during  1996/97.  During  this  process  the  MET  (which 
had  been  invited  to  become  a partner  from  the  beginning)  slowly 
became  formally  involved  and  the  standards  are  now  in  the 
process  of  refinement  with  a view  to  their  becoming  an  official 
provincial  policy  document.  The  goal  is  to  ensure  that  they  repre- 
sent the  best  that  can  be  produced  by  those  most  experienced 
and  knowledgeable  and  thus  be  “authoritative”  in  the  professional 
sense  while  also  being  accepted  and  approved  by  the  MET  and 
thus  acquire  formal  authority  as  well. 

This  mode  of  operation  represents  a departure  from  normal 
procedures  in  Ontario  for  the  development  of  curriculum  policy 
and  is  intended  to  be  an  improvement  on  these  traditional  proce- 
dures. If  standards  are  of  high  quality  but  lack  formal  approval  they 
will  lack  one  kind  of  authority.  If  they  have  formal  approval  but 
lack  professional  respect  they  will  lack  the  other  kind.  In  these 
times  of  shrinking  resources,  we  cannot  afford  either  of  these 
partial  successes.  New  forms  of  partnerships,  such  as  the  ASAP 
partnership,  will  be  required  in  the  future  to  ensure  that  standards 
are  developed  that  have  both.  Our  students  deserve  no  less.  El 
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Robert  Pirsig,  in  his  classic  book  about  quality,  Zen  and  the  Art 
of  Motorcycle  Maintenance,  noted  that  “there  is  no  manual  that 
deals  with  the  real  business  of  motorcycle  maintenance,  the  most 
important  aspect  of  all  — caring  about  what  you  are  doing ” (p.25). 

This  wisdom  transfers  easily  to  many  aspects  of  education. 
Caring  is  the  fundamental  motivation  that  drives  all  of  us  to  take 
responsibility  and  to  dedicate  energy  to  a task  and  see  it  through. 
As  Ontario  completes  its  first  round  of  large-scale  assessment, 
the  responsibility  for  using  the  results  becomes  a shared  one. 
Certainly,  the  Education  Quality  and  Accountability  Office 
continues  to  communicate  what  it  has  learned  and  make  recom- 
mendations for  action.  In  addition,  trustees,  supervisory  officers, 
principals,  and  teachers  in  Ontario  schools  will  take  on  this 
role  in  their  local  communities.  Assessments,  especially  large-scale 
assessments,  can  have  many  purposes  and  can  be  used  (and 
misused)  in  a whole  variety  of  ways.  Educators  in  local  schools 
and  communities  are  the  ones  who  have  responsibility  for  inter- 
preting and  using  the  assessment  results  wisely  and  humanely, 
in  the  service  of  their  students.  They  are  the  ones  who  must 
care  about  the  assessment  and  what  they  are  doing  with  it. 

EQAO  reports  include  achievement  data  and  as  much  contex- 
tual information  as  possible  so  that  they  can  be  interpreted  within 
this  framework.  The  next  stage  is  the  critical  one;  it’s  here  that 
decisions  and  changes  can  contribute  to  enhanced  learning  for 
students  in  Ontario  schools.  We  believe  that  it  is  fair  to  say  that 
no  school,  or  class,  or  teacher,  or  student  has  reached  the  limits  of 
learning  and  growing.  In  every  school  there  are  innumerable  ways 
to  improve  students’  learning  and  to  make  their  learning  more 


efficient  and  more  lasting.  The  results  from  large-scale  assessments, 
like  the  EQAO  Grade  3 provincial  assessment,  can  provide  some 
important  information  to  the  people  who  are  planning  directions 
for  the  school.  School  personnel  and  school  councils  have  the 
task  of  deciding  what  the  results  suggest  and  determining  the 
strategies  that  make  sense  within  the  particular  school  and  the 
community  it  serves.  This  process  has  already  begun  with  the 
specific  and  general  recommendations  that  are  included  in 
the  EQAO’s  Report  of  Provincial  Results. 

EQAO  has  tried  to  set  the  stage  for  Ontario’s  educators  to 
use  the  provincial  large-scale  assessments  as  a starting  point  for 
program  changes  and  improvements  in  learning.  When  EQAO 
began  its  work,  it  determined  that  large-scale  assessment  in 
Ontario  must  be  designed  intentionally  to  provide  high  quality 
information  for  accountability  and  to  foster  genuine  improve- 
ment in  learning  for  all  students.  This  commitment  led  to  two 

Quality  Control 

• EQAO  assessments  reflect  all  aspects  of  the 
provincial  curriculum 

• EQAO  assessments  model  good  instruction  and  good 
assessment  approaches 

• all  materials  and  procedures  were  field-tested,  revised,  and 
field-tested  again 

• scoring  was  systematically  monitored  by  double-marking, 
marking  in  pairs,  discussing  criteria,  blind  marking  of 
selected  papers,  etc.,  and  divergent  markers  were  retrained 


Capacity-building 

• March,  1996  — Joan  Green,  CEO,  undertook  a provincial 
tour  to  meet  with  teachers  and  principals  from  all  boards 

• Ontario  teachers  were  selected  to  work  on  steering  com- 
mittees and  to  be  writers  for  the  assessment  units  or  tasks 

• EQAO  issued  newsletters  and  set  up  a website  to 
provide  information  about  its  activities  and  invite  responses 

• EQAO  provided  preparation  activities  and  example 
materials  for  teachers  and  students  to  familiarize  them 
with  the  assessment  approach 

• all  teachers  and  principals  received  implementation 
training 


vital  criteria  for  the  work  of  EQAO:  every  assessment  must  be 
high  quality  to  ensure  reliable  and  valid  data  on  student  achieve- 
ment, and  the  assessment  must  contribute  to  student  learning  by 
enhancing  the  knowledge  and  skills  of  teachers,  the  educational 
community  as  a whole,  and  the  public. 

Throughout  the  development,  implementation,  and  reporting 
phases  of  the  1996/97  provincial  assessments  (Grade  3 reading/ 
writing/ mathematics  and  Grade  6 mathematics) , EQAO  worked 
hard  to  adhere  to  these  self-imposed  conditions. 

High  Quality  Assessment:  At  every  stage,  EQAO  has  applied 
quality  control  measures  and  procedures  to  monitor  the  quality  of 
the  process  and  it  has  undertaken  a number  of  research  studies  to 
ensure  validity,  reliability  and  comparability  of  the  assessments. 

Contributing  to  Improvement:  EQAO  has  focused  on  the 
improvement  criteria  by  maintaining  contact  and  sharing  ideas 
with  large  numbers  of  stakeholders  and  intentionally  embedding 
capacity  building  activities  into  all  stages  of  its  work.  Educational 
change  and  improved  student  learning  are  internal  processes  that 
happen  in  the  hearts  and  minds  of  the  people  who  live  and  work 
in  classrooms  (teachers  and  students).  One  way  of  influencing 
what  happens  in  classrooms  it  to  provide  teachers  and  administra- 
tors with  information  about  and  training  in  assessment  procedures 
and  uses. 

As  a result  of  the  EQAO  attention  to  quality  and  to  capacity- 
building  for  improvement,  schools  and  boards  have  received  sum- 
maries that  give  good  descriptions  of  the  achievement  of  the 
Grade  3 students  for  whom  the  schools  submitted  booklets  and 
of  some  of  the  contextual  information  that  is  useful  in  interpret- 
ing the  results  and  using  them  for  decision-making.  The  real  work 
follows  when  principals,  teachers,  parent  councils,  and  interested 
community  members  consider  the  results  to  determine  what  they 
mean  and  what  changes  they  might  imply.  Interpreting  assessment 
results  is  not  a mechanical  activity;  it  is  a thinking  process  that 
requires  people  to  reserve  judgment,  put  aside  biases  and  personal 
beliefs,  and  engage  in  a genuine  conversation  about  how  these 
results  can  contribute  productively  to  school  planning. 

When  principals  in  Ontario  schools  received  their  school’s 


Grade  3 data  summaries  from  EQAO  in  October,  1997,  the 
guidelines  for  reporting  cautioned  them:  no  whining;  no  gloating. 
Principals  and  teachers  are  the  front-line  interpreters,  users,  and 
communicators  of  the  assessment  results.  Whining  about  the 
results,  or  worse  about  the  students  who  were  assessed,  serves  no 
productive  process.  Neither  does  gloating  about  the  quality  of  the 
school  when  the  students’  achievement  is  a product  of  many  fac- 
tors. These  cautions  are  based  in  the  recognition  that  understand- 
ing the  data  from  any  large-scale  assessment  involves  considering 
all  of  the  variables  that  can  influence  assessment  results.  Most  of 
these  variables  cannot  be  controlled  for  but  are  important  for 
planning.  They  include  dynamics  like  the  socio-economic  status 
of  the  students,  contextual  activities  like  events  in  the  life  of  the 
school,  and  other  conditions  that  are  impinging  on  the  lives  of 
individual  students  (e.g.,  family  issues).  At  the  same  time,  there  are 
many  conditions  that  can  affect  student  achievement  that  schools 
can  influence;  things  like,  instructional  emphasis,  school  organiza- 
tion, and  availability  of  resources. 

The  following  figure  shows  the  cyclic  relationship  that  should 
exist  between  assessment  and  learning. 


Assessment  as  Part  of  Learning 
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classroom  assessment,  and  student  motivation 

Linda  Darling-Hammond  has  identified  the  following  set  of 
questions  for  principals  and  teachers: 

• What  do  we  want  our  students  to  be  able  to  do? 

• How  will  we  know  whether  our  students  can  do  these  things? 

• What  measures  can  we  develop  to  evaluate  their  knowledge 
and  abilities? 

• How  do  we  develop  shared  views  of  what  constitutes 
competence? 

• How  will  we  help  our  students  get  there? 

These  questions  can  be  a starting  point  for  the  conversations 
that  occur  in  schools  all  over  the  province,  as  they  consider  the 
Grade  3 results  and  the  contextual  information  that  surrounds 
them.  The  conversations  will  not  be  brief  and  they  will  inevitably 
be  emotional,  but  they  hold  the  promise  of  engaging  educators 
and  the  community  in  discussion  about  what  is  important  in 
education  and  about  the  roles  that  each  of  them  can  play  in  help- 
ing young  people  perform  the  best  that  they  can.  El 
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Performance-Based 

Standards 


A "T"  a time  when  there  is  little  agreement  about  what 
/\  I should  be  taught,  educational  institutions  are 
being  flooded  with  increasing  demands  for  accountability 
and  for  raising  standards.  This  situation  may  stem  from 
the  fact  that  we  are  living  between  two  eras.  The  people 
who  are  making  demands  were  born,  educated  and 
worked  in  the  industrial  era;  their  children,  those  in 
school  today,  are  preparing  for  the  information  age. 

The  education  that  was  good  enough  for  their  parents 
is  not  good  enough  for  them. 

The  Etobicoke  Board  of  Education  has  found  that 
using  performance-based  assessments  and  having 
clear  standards  to  assess  those  performances  is  one 
constructive  strategy  for  bridging  this  gap. 


Why  does  performance-based 
assessment  help? 

The  information  age  demands  the  mastery 
of  what  educators  call  process  skills.  People 
must  be  able  to  identify  questions  or 
problems,  find  and  process  relevant  infor- 
mation, draw  and  present  a conclusion, 
and  go  on  to  ask  another  question. 
Nowhere  in  this  list  of  skills  is  there  an 
expectation  for  mastery  of  the  simple 
information  that  has  traditionally  been 
the  focus  for  education  and  testing.  This 
shift  from  mastery  of  information  to  mas- 
tery of  skills  for  processing  information  is 
a major  one. 

Performance-based  assessment  can  be 
an  important  tool  to  assess  the  mastery  of 
these  process  skills.  It  allows  educators  to 
give  students  problems  similar  to  what 
they  might  be  called  upon  to  solve  in  gen- 
uine situations  and  to  assess  whether  they 
can  carry  out  the  steps  required  for  prob- 
lem-solving. Teachers  can  both  teach  and 
do  assessments  using  problem-solving 
challenges.  Such  tasks  allow  learners  to  act 
as  committed,  participating  members  of  a 
community  of  problem-solvers  where 
their  contributions  are  truly  valued. 
Underlying  this  strategy  is  the  knowledge 
that  adults  who  have  “real  jobs,”  and  make 
“real  life  decisions,”  must  feel  this  same 
sense  of  commitment  in  order  to  make  the 
real  world  function. 

In  performance-based  activities,  or 
assessments,  students  can  practice  “real 
world”  involvement  as  well  as  learn  and 
demonstrate  academic  skills  and  knowl- 
edge. Their  performance  provides  informa- 
tion to  them  and  to  the  teacher  about 
what  is  needed  for  improvement  and  it 
allows  students  to  become  active  partici- 
pants in  their  own  education. 


What  are  standards  and  rubrics? 

When  performance-based  assessments  are 
used  in  a broader  context  to  meet 
accountability  demands,  the  issue  of  stan- 
dards becomes  crucial.  There  are  two  key 
meanings  of  standards  that  must  be  distin- 
guished. In  the  first  sense,  standard  means  a 
model  of,  or  for,  student  performance.  It 
involves  describing  what  students  do  or 
look  like  when  carrying  out  different 
tasks.  In  the  second  sense,  standard  means  a 
measure  of  quality.  In  this  sense,  a rubric, 
or  descriptive  scoring  scale,  must  be  devel- 
oped that  clearly  specifies  the  different  lev- 
els of  achievement  and  it  must  be  applied 
to  tasks  that  are  developmentally  appropri- 
ate at  each  grade.  In  Etobicoke  we  have 
found  the  latter  sense  of  standards  to  be 
more  useful. 

A standard  as  a measure  of  quality  is 
defined  by  the  rubric,  or  scale,  that  speci- 
fies the  different  levels  of  achievement.  In 
order  for  a rubric  to  be  a good  measuring 
instrument  and  lead  to  reliable  scores  it 
must  have  the  following  characteristics: 

• the  components  of  each  rubric  must 
be  clearly  specified 

• each  component  must  appear  in  each 
level  of  the  rubric  (i.e.,  if  accuracy  is 
included  in  the  description  of  Level  1,  it 
must  be  in  Levels  2,  3,  4,  5 and  6 also) 

• the  levels  of  the  rubric  must  be  as  near- 
ly equidistant  as  possible  (i.e.,  the  dis- 
tance between  Level  1 and  2 should  be 
the  same  as  the  distance  between  Level 
3 and  4) 

One  of  the  greatest  benefits  of  evaluat- 
ing student  work  using  a rubric  is  that  the 
standards  represented  by  the  rubric  can  be 
made  public.  The  rubric  can  be  shared 
with  students,  with  parents  and  with  the 
community  so  that  they  can  see  the  crite- 
ria of  a good  performance.  This  means  that 
students  can  concentrate  on  learning  the 
subject,  rather  than  simply  focusing  on 
what  they  believe  the  teacher  will  value. 
For  students,  rubrics  can  be  a component 
of  personal  growth  and  a tool  for  lifelong 
learning.  Useful  rubrics  are  clear  and  con- 
sistent across  grades  and  tasks  so  that,  once 
learned,  they  can  be  applied  to  new  tasks 


at  each  succeeding  grade.  This  means  that 
there  can  be  a single  rubric  for  each  criti- 
cal component  of  major  categories  of 
tasks.  An  example  of  a critical  component 
would  be  writing.  A writing  rubric  can 
stay  the  same  across  all  grades.  For  each 
grade,  it  will  be  applied  to  tasks  that  vary 
in  difficulty  with  the  age  of  the  learner. 
This  is  entirely  possible  if  the  rubric  relates 
to  the  components  of  doing  a job  well. 

In  the  Etobicoke  Board  of  Education, 
we  administer  performance-based  assess- 
ments in  both  language  and  mathematics. 
For  the  language  assessments,  we  use  three 
rubrics:  Reading  Comprehension,  Writing 
Performance,  and  Language  Usage.  Writing 
Peformance  looks  at  the  organization, 
focus,  and  development  of  the  piece  of 
writing  and  the  overall  accomplishment  of 
the  task  assigned.  This  is  the  creative, 
thoughtful  side  of  writing.  Students  must 
present  their  ideas  and  support  them  with 
information  from  their  reading  or  in  fic- 
tion writing  they  must  pick  up  characters 
and  carry  them  forward  appropriately. 

They  must  also  show  evidence  of  using  a 
logical  plan  in  their  writing.  When  stu- 
dents can  take  a piece  of  writing  and  say 
that  it  was  good  because  ideas  were  pre- 
sented clearly  but  it  would  be  better  if 
there  were  more  supporting  information, 
it  is  clear  that  the  student  is  learning  the 
important  elements  of  writing.  By  using 
the  rubric  to  score  all  writing,  whether  for 
English,  social  studies,  or  science,  there  is 
continuity  in  the  collection  of  assessment 
information  and  the  same  standards  are 
being  applied  to  all  demonstrations  of  the 
same  set  of  skills. 

How  do  standards  relate 
to  the  age  of  the  student? 

Within  a model  that  uses  a single  rubric  to 
score  performances  at  all  grades,  standards 
are  defined  in  terms  of  the  levels  of  per- 
formance that  are  described  by  the  rubric. 
The  rubric  interacts  with  an  age/grade 
continuum  that  specifies  tasks  and  materi- 
als that  are  appropriate  for  learners  at  each 
grade.  The  task  given  to  a Grade  3 child 
and  one  given  to  a Grade  8 student  may 
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be  the  same  (e.g.,  to  write  a report)  but 
the  complexity  of  the  issues,  the  reading 
material,  the  vocabulary,  and  the  concepts 
will  differ  greatly  for  the  two  grades.  The 
performance  required  to  attain  each  level 
of  the  rubric  will  differ  between  the  two 
grades  as  well.  Samples  of  student  work 
that  demonstrate  performance  at  each  level 
of  the  rubric  at  each  grade  are  useful  to 
show  precisely  what  should  be  expected. 

In  Etobicoke,  the  performance-based 
assessments  use  tasks  at  each  grade  which 
have  been  chosen  to  be  appropriate  to  the 
ages  and  stages  of  learners.  These  tasks 
must  be  open-ended  to  allow  learners  to 
perform  at  different  levels  rather  than  hav- 
ing tasks  that  are  assessed  at  different  levels 
of  difficulty  within  a grade. 

We  have  been  speaking  of  a model 
where  rubrics  define  a standard.  Within  a 
rubric  there  are  several  levels,  commonly 
4,  5,  or  6.  Frequently  the  word  standards  is 
used  to  specify  the  level  on  the  rubric  that 
must  be  reached  in  order  for  the  perfor- 
mance to  be  considered  acceptable.The 
Education,  Quality  and  Accountability 
Office  used  a 4-pomt  rubric,  as  specified 
by  the  Ministry  of  Education  and  Training, 
to  score  the  1996-1997  Grade  3 and  6 
performance-based  tests.  Based  on  the 
Common  Curriculum,  Eevels  2,  3 and  4 were 
defined  as  “at  grade  level.” 

One  could  “raise  standards”  by  increas- 
ing the  level  that  must  be  attained  for  per- 
formance to  be  considered  acceptable.  For 
example,  considering  Level  2 as  below 
grade  level  raises  the  standard.  Raising  the 
standard  in  this  way  will  lower  the  propor- 
tion of  students  who  meet  the  standard 
until  changes  in  program  can  help  students 
who  have  been  performing  at  Level  3 


attain  a higher  level. 

Another  way  of  raising  standards  in 
this  model  is  to  increase  the  difficulty  of 
the  tasks  administered  at  each  grade.  This 
can  happen  in  a variety  of  ways.  It  may 
involve  increasing  the  overall  difficulty  of 
the  task,  requiring  a type  of  writing  that 
is  not  often  taught,  or  covering  a strand 
of  mathematics  that  is  not  often  a focus. 

Why  performance-based 
assessments  and  standards? 

If  we  must  shift  our  teaching  to  cover  the 
skills  demanded  by  the  information  age, 
we  need  a way  of  assessing  student  perfor- 
mance so  that  we  can  know  if  we  are  suc- 
cessful. We  need  a way  of  looking  across 
our  program  to  see  what  areas  are  being 
taught  successfully  and  what  areas  need 
improvement.  We  need  a way  of  meeting 
demands  for  accountability  that  is  consis- 
tent with  good  programming,  that  tests 
what  we  are  actually  teaching,  and  that 
provides  a learning  experience  for  students 
while  providing  high  quality  assessment 
information.  Performance-based  assess- 
ment provides  all  of  this. 

Performance-based  assessment  is  useful 
for  reporting  to  parents  because  the  results 
are  clear  and  concrete.  The  actual  piece 
of  work  can  be  shown  to  a parent,  and 
the  rubric  makes  clear  the  criteria  that 
have  been  used  to  score  it.  The  rubric  also 
helps  identify  the  next  steps  for  instruction 
and  practice. 

The  results  of  performance-based 
assessment  can  be  used  by  an  entire  school 
staff  to  change  both  instruction  and  pro- 
gram to  improve  student  performance.  In 
addition,  by  looking  at  the  strengths  and 
weaknesses  in  performance  as  revealed  by 
many  students  on  many  different  tasks  it 
is  possible  to  assess  the  consistency  of 
curriculum  coverage  across  grades.  When 
many  performance  activities  are  taken 
together,  all  curriculum  elements  can  be 
covered  and  one  can  see  where  improve- 
ment is  needed. 

For  example,  using  our  performance- 
based  assessment  results,  we  learned  that 
we  were  doing  a better  job  of  teaching 


fiction  reading  and  writing  than  nonfic- 
tion reading  and  writing.  This  has  resulted 
in  some  concrete  decisions  in  schools  — 
for  example,  using  available  money 
to  increase  the  nonfiction  collections  in 
school  libraries.  Even  more  substantial 
plans  were  developed  by  several  schools 
as  a response  to  these  results.  Some  are 
implementing  the  First  Steps  language 
program,  some  a program  called  the 
Literacy  Diet,  and  some  have  chosen 
other  initiatives  to  improve  their  students’ 
achievement  in  all  aspects  of  language. 
These  staff  development  plans  are  focused 
on  enabling  teachers  to  make  specific 
changes  to  program. 

When  we  begin  making  changes,  we 
find  they  connect  in  unexpected  ways  to 
unrecognized  areas  of  need.  For  example, 
there  is  research  showing  that  girls  prefer 
reading  fiction  and  that  boys  prefer  reading 
nonfiction  and  other  research  showing  that 
girls  do  not  do  as  well  on  mathematics  as 
boys.  In  performance  assessments  that  cover 
all  types  of  reading  and  writing  we  are 
balancing  the  program  to  make  it  more 
consistent  with  the  interests  of  boys  and  to 
help  girls  develop  skills  they  will  need  in  the 
workplace.  By  including  communication 
as  an  area  for  assessment  m mathematics  we 
are  making  the  program  more  consistent 
with  the  interests  of  girls  and  helping  boys 
to  develop  needed  skills.  In  both  cases, 
student  achievement  should  improve. 

Performance-based  assessments  using 
simple,  clear  rubrics  that  spell  out  stan- 
dards that  can  be  applied  to  a wide  range 
of  tasks  across  all  grade  levels  can  lead  to 
true  system  renewal.  Changes  that  grow 
out  of  the  experience  of  teachers  openly 
and  honestly  looking  at  the  results  of  their 
students  in  coping  with  challenging  tasks 
truly  are  changes  that  lead  to  improvement 
and  raise  standards. 

Perhaps  the  best  summary  of  the  pur- 
pose of  performance-based  assessment  in 
our  system  came  from  a Grade  5 student 
who  said,  “I  know  why  you  gave  us  this 
test.  When  we  grow  up  we  will  have  to  write 
reports  and  you  want  to  know  if  we  are  learn- 
ing how.  ” Exactly.  El 
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the  very  nature  of  their  work,  teachers  tend  to  work  in 
isolation.  Rarely  do  they  articulate  for  themselves  or 
to  their  colleagues  what  they  do  or  compare  their  practices  with 
those  of  others.  It  is  no  different  in  the  case  of  evaluation.  The 
result  is  often  that  teachers  evaluate  quite  differently  from 
each  other  and  students  have  to  guess,  for  each  teacher,  what 
the  criteria  for  a successful  performance  are. 

One  process  we  have  found  useful  in  the  Carleton  Board  of 
Education  to  break  down  this  isolation  is  what  we  call  “confer- 
ence-marking” or  “teacher  moderation.”  Moderation  of  student 
work  is  a common  enough  practice.  Typically,  expert  assessors 
external  to  the  academic  institution  — university,  professional 
school,  or  school  board  — review  a sample  of  students’  work 
marked  by  local  teachers.  If  the  external  moderators  do  not 
agree  with  the  internal  evaluation,  they  may  substitute  their 
evaluation  for  it,  or  recommend  or  even  order  the  internal 
marker  to  adjust  marks. 

The  advantage  of  external  moderators  is  that  they  provide  a 
sense  of  “objectivity”  and  of  standards.  The  sample  work  they 
are  assessing  comes  from  all  teachers  in  the  system,  so  that  the 
moderators  are  applying  the  same  standards  across  the  board. 
Moreover,  since  the  external  moderators  are  not  personally 
involved  with  the  students  being  evaluated,  there  can  be  no 
question  of  unfair  treatment  of  students  or  of  standards  being 
adjusted  for  some  particular  purpose. 

There  are  also  disadvantages  with  this  process.  First,  external 
judgments  may  be  based  on  standards  and  criteria  just  as  unartic- 
ulated as  those  of  the  classroom  teacher.  There  is  simply  a differ- 


ent set  of  implicit  preferences.  In  this  case,  local  teachers  resent  the 
recommendations  — probably  rightly  — as  arbitrary  and,  since 
the  rationale  is  not  explicated,  teachers  do  not  improve  their  own 
skills  in  evaluating.  Consequently,  while  the  external  moderators 
may  provide  some  consistency  across  students  in  evaluating  the 
product  of  one  event,  the  evaluations  which  students  receive 
for  the  rest  of  the  course  continue  to  reflect  any  inconsistency 
there  may  be  among  teachers.  Second,  the  teacher’s  professional 
function  has  been  usurped.  It  is  the  teacher  who  is  answerable  to 
the  student,  not  the  external  assessor.  It  is  difficult,  if  not  impossi- 
ble, at  times,  for  the  teacher  to  defend  the  “official”  evaluation  to 
the  student. 

In  Carleton,  we  believe  that  conference-marking  achieves 
what  external  moderation  is  intended  to  achieve  while  avoiding 
the  problems. 

Teachers  of  a particular  course  get  together,  bringing  with 
them  samples  of  their  students’  work,  which  they  have  already 
marked.  At  the  conference-marking  session,  they  re-mark  work 
produced  by  students  of  other  teachers.  Then,  the  student’s  own 
teacher  and  a second  marker  compare  their  evaluations.  The  two 
teachers  examine  the  student’s  work  closely  and  talk  about  why 
they  have  marked  the  piece  of  work  as  they  have.  Of  course,  the 
discussion  is  more  productive  if  there  is  a significant  difference 
between  the  two  teachers’  marking  and  they  try  to  reach  agree- 
ment as  to  what  the  final  mark  should  be.  If  they  cannot  agree,  the 
paper  is  marked  by  a third  teacher  who  then  joins  the  conference. 

Conference-marking  achieves  a number  of  things.  For  the 
student,  it  means  greater  fairness.  His  or  her  teacher’s  evaluations 
have  been  considered  by  a second  and,  sometimes,  a third  teacher. 
For  the  teacher,  it  means  support.  The  teacher  returns  papers  to 
students  confident  that  his  or  her  evaluations  have  been  reviewed 
by  a colleague. 

Most  importantly,  conference-marking  provides  teachers  with 
a forum,  first  in  individual  conferences  with  one  or  two  other 
teachers,  and,  later,  in  general  discussion  with  all  teachers  present. 


to  talk  about  their  criteria  and  standards.  Teachers  have  an  oppor- 
tunity to  articulate  what  they  believe  and  to  hear  what  other 
teachers  believe.  As  a result  of  this  discussion,  teachers  adjust  their 
evaluations  and  become  more  consistent  within  the  system  in 
their  criteria  and  standards,  and  their  application. 

This  sort  of  teacher-moderation  provides  opportunities  for 
articulation  and  reflection  in  several  ways.  In  comparing  their 
marking  of  a student’s  work,  teachers  have  to  talk  about  why  they 
made  the  judgment  they  did;  consequently,  they  have  to  make 
explicit  the  assumptions,  standards,  and  constructs  that  underlie  the 
judgments  they  make  either  to  defend  them  or  to  change  them. 

By  participating  in  conference-marking,  teachers  become 
aware  of  the  objectives  that  are  being  evaluated  by  other  teachers. 
Inevitably,  the  teacher  must  compare  these  objectives  with  those 
that  operate  in  his  or  her  own  classroom.  Exemplars  of  student 
work  illustrating  the  objectives  lead  teachers  to  compare  their 
own  expectations  for  student  performance  with  the  exemplars. 
Above  all,  teachers  see  work  of  students  other  than  their  own  and 
thus  receive  feedback  in  terms  of  how  these  students  in  other 
classrooms  are  doing  relative  to  their  own  students. 

Finally,  conference-marking  leads  to  public  discussion  and  it  is 
in  public  discussion  that  teachers  must  confront  the  diversity  of 
their  colleagues’  knowledge,  beliefs,  and  practices. 

A Sketch  of  the  Process 

The  first  stage  in  teachers’  development  is  awareness,  a realization 
among  teachers  of  the  diversity  and  similarities  which  exist 
among  their  colleagues.  The  awakening  of  such  awareness  is  seen 
frequently  in  workshops  held  for  teachers  in  the  early  stages  of  a 
project.  An  opening  activity  in  such  workshops  is  one  in  which 
teachers  mark  the  same  piece  of  student  work,  first  alone  and 
then  in  groups  with  other  teachers.  An  inevitable  outcome  of 
such  an  exercise  is  surprise  at  the  wide  range  of  marks  the  piece 
receives  and  the  varying  rationales  that  are  used  to  justify  these 
marks.  This  phenomenon  has  been  observed  equally  in  English, 
mathematics  or  science  and  seems  to  apply  whether  the  student 
work  is  an  essay,  an  open -response,  as  in  a solution  to  a problem 
in  mathematics,  or  even  a multiple-choice  question. 

The  second  stage  is  clarification  when  teachers  make  explicit 
what  in  practice  is  implicit  and  unarticulated.  In  discussions  with 
other  teachers,  they  attempt  to  explain  why  they  evaluated  the 
student  work  in  the  way  they  did.  Gradually  they  uncover  their 
hitherto  implicit  criteria.  Frequently,  participants  discover  that 
they  are  using  different  criteria  from  those  of  their  colleagues  and 
an  important  discussion  concerns  the  significance  of  the  different 
criteria.  Moreover,  teachers  come  to  realize  that  they  may  not  be 
talking  about  the  same  thing,  even  though  they  are  using  the 
same  words. 

Early  in  the  development  of  Carleton’s  OAC  I English  exami- 
nation, a heated  debate  took  place  on  how  much  the  element 
ol  “support"  should  be  worth.  During  the  debate,  participants 


realized  that  different  conceptions  of  support  were  being  dis- 
cussed. Some  teachers  wanted  students  to  provide  as  much  detail 
from  the  text  as  possible;  others  assumed  that  the  student’s  evi- 
dence should  be  selective  and  limited.  Furthermore,  some  teach- 
ers assumed  that  support  involved  simply  the  factual  presentation 
ot  evidence;  others  assumed  that  it  involved  the  interpretation 
ot  evidence  and  its  integration  into  the  argument  and  organiza- 
tion of  the  essay.  Adherents  of  each  different  notion  of  support 
believed  that  everyone  else  shared  their  point  of  view. 

The  third  stage,  which  is  not  inevitable,  is  synthesis  or  agreement, 
by  which  teachers  in  a jurisdiction  establish  a common  set  of  cri- 
teria and  standards.  Some  issues  prove  to  quite  divisive.  For  exam- 
ple, among  English  teachers,  mechanics  — grammar,  punctuation, 
spelling  — is  such  an  issue.  Some  teachers  argued  that  every  error 
should  be  recorded  and  some  even  argued  that  a piece  of  work 
which  did  not  pass  in  mechanics  should  not  be  given  a passing 
mark;  other  teachers  argued  that  such  errors  should  be  evaluated 
in  terms  of  the  way  such  errors  interfere  with  communication  in 
the  student  s work.  Among  mathematics  teachers,  the  conception 
of  the  subject-matter  is  an  issue.  There  are  teachers  who  conceive 
of  mathematics  as  a set  of  discrete  skills  which  are  to  be  applied 
appropriately;  on  the  other  hand,  there  are  those  who  conceive  of 
mathematics  as  a means  for  solving  problems. 

Even  if  the  agreement  is  to  beg  to  differ,  at  least  teachers  are 
now  aware  (a)  of  their  own  evaluation  criteria  and  (b)  of  the 
variations  in  the  system.  They  are  in  a position  to  make  these 
publicly  known  to  their  students. 

Flowever,  in  Carleton  we  have  done  better  than  that.  Teachers 
felt  that  criteria  for  successful  work  should  be  set  out  in  a book 
along  with  “exemplars"  of  such  work  and  that  the  book  should 
be  made  available  to  students.  Exemplar  Booklets,  in  one  form 
or  another,  are  now  available  in  intermediate  English,  senior 
English,  mathematics,  chemistry,  and  biology. 

A publicly  articulated  set  of  criteria  for  a successful  perfor- 
mance and  exemplars  has  the  following  benefits: 

• Teachers  have  a clearer  knowledge  of  the  salient  features  of  a 
task  and  are,  therefore,  able  to  be  more  focussed  in  their  teach- 
ing and  to  be  more  explicit  to  students  about  their  expectations. 

• The  corollary  is  that  students  now  have  clearly  stated  descrip- 
tions of  what  a task  requires;  they  also  know  what  they  have 
to  do  to  improve. 

• The  result  is  that  student  product  improves. 

• Students  have  confidence  that  teachers  are  more  consistent  in 
their  marking. 

• Teachers  and  students  are  much  more  able  to  talk  about  a 
student’s  work,  focusing  on  elements  in  the  work  compared 
with  the  criteria,  rather  than  with  personal  preferences  of 
teachers  or  students.  In  particular,  teachers  are  able  to  show 
students  what  they  have  to  do  to  improve.  Similarly,  group- 
work  and  peer  marking  are  facilitated. 

• Finally,  parents  are  able  to  see,  in  some  detail,  what  is  expected 


of  their  children.  Not  only  is  the  frustration  of  vague  expecta- 
tions removed,  but  parents  see  that  the  tasks  and  expectations 
are  far  from  trivial.  Parents  are  also  more  confident  in  the 
consistency  of  marking  from  teacher  to  teacher  and  from 
school  to  school.  El 

Note:  David  Ireland  can  be  reached  at 

167  MacKay  Street,  Ottawa,  Ont.,  KIM  2B5. 
ph.  (613)  744-4345,  e-mail  direland@cyberus.ca 

Moderation:  A Different 
Perspective  on  Standards 

Susan  Elliott 

Teacher,  Toronto  Board  of  Education 

f ould  you  do  a re-mark  on  this  history  exam? ’’Remarking 
a major  final  exam  can  make  a teacher  quite  nervous.  The 
student  is  obviously  unhappy  with  the  grade,  and  has  asked  for  a 
second  opinion. The  result  could  make  the  difference  between 
going  on  to  university  or  not.  As  a teacher,  1 often  worry  about 
these  considerations.  I also  wonder  about  the  differences  in  mark- 
ing standards  between  teachers.  Of  course,  exams  are  set  by  a 
team  of  teachers  according  to  the  format  and  level  of  difficulty 
required.  A detailed  marking  scheme  is  also  made  and  shared.  But 
variances  in  grading  in  areas  such  as  writing  style,  development  of 
argument,  and  choice  of  examples  and  support  still  leave  wide 
berth  for  individual  marking  differences.  If  I grade  the  paper 
higher,  what  does  that  say  about  the  other  teacher’s  criteria  for 
marking?  What  does  that  say  about  my  own  standards?  What  does 
it  say  about  the  “objective”  evaluation  of  student  work? 

The  Search  for  Standards 

From  elementary  school  to  university,  the  problem  of  setting  and 
maintaining  some  consistency  in  standards  of  evaluation  has 
sparked  research  and  the  development  of  a wide  array  of  possible 
solutions.  Outcome-based  curriculum  design  has  dominated  the 
curriculum  field  for  several  years  now  in  an  effort  to  make  clear 
the  criteria  for  achievement.  Curriculum-based  assessment 
schemes  such  as  “Benchmarks”at  the  Toronto  Board  of  Education 
also  attempt  to  train  teachers  to  evaluate  according  to  set  out- 
comes and  standards,  instead  of  basing  judgments  on  unarticulated 
professional  expertise.  The  use  of  standardized  assessments  evaluat- 
ed by  trained  markers  is  another  attempt  to  gain  “quality  assur- 
ance” in  educational  assessment  (Gipps,  1994).  All  of  these  are 
expensive  and  time-consuming  initiatives  whose  efficacy  is  still 
debated.  While  doing  graduate  work  in  London,  England,  I began 
to  study  and  observe  yet  another  method, with  a long  history  of 
use  at  the  secondary  level,  called  “moderation.”  In  1994,  modera- 
tion was  beginning  to  be  used  in  primary  schools  as  a result  of 
the  1990  National  Curriculum  and  assessment  arrangements. 


Defining  Moderation 

To  moderate  means  to  balance,  to  even  out,  and  to  eradicate 
extremes.  For  example,  we  use  the  word  modify  when  we  talk  of 
water  having  a modifying  effect  on  the  land  nearby;  making  the 
climate  less  cold  or  hot.  Moderation  has  a variety  of  forms 
(Elarlen,  1994).  Some  of  these  include  statistical  moderation  using 
scaling  techniques  to  account  for  individual  marking  differences 
between  teachers  on  tests.  But  of  interest  here  is  what  is  termed 
“consensus  moderation”  (Nuttall,  1993)  or  “social  moderation.” 
Simply  stated,  moderation  involves  teachers  getting  together  to 
talk  and  assess  real  student  work  using  exemplars,  curriculum  out- 
comes, and,  if  available,  assessment  levels  and  criteria.  The  purpose 
of  the  exercise  is  to  come  to  some  shared  understandings  of  the 
criteria.  Both  the  processes  used  in  the  work  and  the  product  are 
considered.  Sometimes  the  grading  is  done  by  the  teachers 
working  together;  sometimes  work  has  been  already  assessed  and 
is  re-evaluated.  In  all  cases,  moderation  involves  teacher  commu- 
nication and  discussion. 

The  assessment  arrangements  for  the  National  Curriculum  in 
England  and  Wales  were  outlined  in  the  report  made  by  The  Task 
Group  of  Assessment  and  Testing.  The  report  established  the  two 
main  functions  of  moderation: 

1.  to  communicate  the  standards  of  the  curriculum 

2.  to  control  deviations  from  the  general  standards  by  making 

appropriate  changes  or  adjustments 

The  report  talks  of  “group  moderation”  where  the  emphasis 
is  on  communication  through  discussion  and  exchange  of  samples 
at  a meeting.  The  process  provides  a setting  for  teachers  to  “clarify 
their  own  judgments  by  having  to  explain  them  to  others  and 
in  doing  so  reveal  the  basis  of  their  assessments”  (Conner,  1991, 
p.  17). 

What  the  Process  Looks  Like  in  Action 

In  several  of  the  schools  I observed,  the  staff  had  appointed  a 
school  assessment  co-ordinator.  Since  the  first  of  the  national 
assessments  occurred  at  Year  Two  (seven-year-olds),  most  co-ordi- 
nators  were  Year  Two  teachers.  These  co-ordinators  had  done 
some  extra  inservice  training  with  the  national  standardized 
assessment  materials  and  were  more  familiar  with  the  curriculum 
and  the  criteria  for  achievement.  At  regular  intervals,  say,  once  a 
term,  part  of  a staff  meeting  was  dedicated  to  moderation  discus- 
sions. The  atmosphere  was  very  informal  and  teachers  met  in 
small  groups  of  five  or  six.  The  co-ordinator  selected  most  of  the 
materials  including  student  work. The  student  was  not  named, 
nor  was  the  student’s  teacher.  Sometimes  the  work  had  come 
from  a student  portfolio  and  was  already  marked.  The  age  and 
grade  of  the  student  was  given  before  the  discussion.  Teachers 
began  to  make  comments  about  what  was  good,  or  not  so  good, 
or  missing  from  the  product.  Some  spoke  about  what  the  child 
must  have  had  to  understand  to  produce  such  a work.  Just  where 
the  assignment  might  fit  into  the  curriculum  outcomes  was 
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sometimes  mentioned  by  the  co-ordinator  as  well,  although  most 
of  the  co-ordinators  were  fairly  new  at  the  post  and  had  to  look 
up  a great  deal  in  the  curriculum  documents. 

Exemplars  from  texts  or  assessment  material  were  used  to 
point  out  what  various  levels  of  achievement  might  look  like. 
After  the  National  Curriculum  was  put  in  place,  materials  and 
texts  became  available  to  teachers.  Included  were  pictures  or 
photocopies  of  student  work  to  be  used  as  exemplars.  At  some 
point  in  the  discussion,  teachers  started  to  say  what  level  or  mark 
they  would  give  to  the  work  and  why.  Others  might  disagree  and 
give  their  own  reasons  for  their  judgments.  At  all  times  the 
atmosphere  was  collegial  because  the  meetings  were  understood 
to  be  inservice  meetings  and  part  of  their  training.  They  were 
in  no  way  seen  as  “high  stakes”  assessment  for  the  children  or  the 
teachers.  For  these  reasons,  teachers  felt  free  to  voice  their  opin- 
ions. It  appeared  at  times  that  the  teachers  with  the  least  teaching 
experience  did  the  most  talking,  but  all  teachers  participated  to 
some  degree. 

The  Advantages  of  Moderation 

A number  of  important  benefits  could  be  gained  from  wide- 
spread use  of  moderation  meetings: 

1 . Establishing  school  standards  through  the  development  of  assessment 
skills  — The  assessment  meetings  produced  a variety  of  samples 
of  student  work  and  teacher  assessments.  Over  time,  the  collec- 
tion should  include  samples  from  all  the  grades  in  the  school  in 

a wide  variety  of  subjects.  It  was  explained  to  me  that,  at  some 
point,  all  the  samples  could  be  collected  into  a school  portfolio. 
The  idea  was  that  a when  a school  as  a whole  was  evaluated, 
part  of  the  inspection  would  include  an  examination  of  the 
school  portfolio.  Evaluators  could  get  a clear  understanding  of 
the  standards  of  the  school  from  examining  how  they  judged  the 
student  work.  Ostensibly,  it  would  be  possible  to  say,  for  example, 
that  the  standards  for  Grade  3 writing  and  editing  were  too  low, 
it  a top  paper  at  one  school  marked  higher  than  the  district  or 
national  standards. 

2.  Iticreased  understanding  of  curriculum  and  achievement  — 
Evidence  from  research  is  widespread  that  teachers  are  not  well 
trained  in  assessment.  Teachers  need  to  become  more  “assessment 
literate” (Gipps,  1994,  p.  160).  Educational  assessment  involves 
grading  or  scoring  by  teachers  or  trained-raters:  in  order  to 
enhance  consistency  of  scoring,  teachers  need  to  understand  the 
scoring  categories  and  the  level  of  performance  associated  with 
them.  This  can  be  achieved  through  a process  of  moderation  and 
provision  ot  exemplars.  The  exemplars  and  standards  can  also 
serve  the  purpose  of  explicating  to  teacher  the  nature  of  the  skill 
or  concepts  being  taught  and  assessed. 

3.  Maintaining  standards  across  schools  — Accountability  and 
quality  control  could  be  maintained  in  way  quite  different  than 
through  national  assessment  scores.  Showing  accountability,  while 
at  the  same  time  providing  inservice  training,  seemed  to  many 


teachers  to  be  an  effective  use  of  their  time.  The  moderation 
group  might  be  made  up  of  teachers  of  one  subject  or  grade 
in  one  school,  or  from  a number  of  schools.  The  meetings 
across  schools  were  valued  for  their  purpose  of  maintaining 
consistency  of  judgments  between  schools.  Once  again,  the 
school  portfolio  could  be  used  to  compare  and  maintain  consis- 
tency of  standards 

4.  Enhancing  the  reliability  of  teachers’  professional  judgments  — 
Moderation  began  in  England  in  the  1960s  and  ’70s  when  high 
school  assessment  changed  from  an  examination-driven  system  to 
graded  assessment,  where  course  work  accounted  for  more  or 
sometimes  all  of  the  final  mark.  When  teachers  were  responsible  for 
all  evaluation,  they  had  to  be  as  articulate  and  accountable  as  possi- 
ble about  their  assessment  systems.  Teachers  had  to  become  highly 
skilled  in  assessment  practices,  using  a variety  of  measures  and 
methods.  Moderation  became  an  important  part  of  that  training. 
Nurrall  (1993)  has  argued  for  "the  power  of  consensus  moderation 
across  schools  in  enhancing  teachers’  understanding  of  the  curricu- 
lum, in  widening  their  horizons  of  teaching  approaches,  teaching 
materials  and  assessment  approaches,  and  in  bringing  them  to  a 
common  and  shared  understanding  of  levels  or  grades.  Consensus 
moderation  is  a very  powerful  form  of  professional  development.” 
By  improving  teachers’  assessment  skills,  more  confidence  and 
credence  can  given  to  their  final  evaluations  about  achievement. 

5.  Establishing  a cost-efficient  assessment  process.  — The  modera- 
tion process  is  inexpensive  as  compared  to  full-scale  testing.  More 
value  can  be  placed  on  teacher  assessments  if  judgments  are  seen 
to  be  objective  and  reliable.  That  being  the  case,  then  less  time, 
energy,  and  money  needs  to  be  spent  on  full-scale  standardized 
assessments  implemented  to  confirm  teacher  judgments. 

One  of  the  main  benefits  to  moderation  seems  to  be  the  ease 
with  which  the  process  can  be  started  and  maintained.  It  is  an 
intensely  social  event  and  engages  teachers  with  material  and  ideas 
vital  to  their  work  in  the  classroom.  It  encourages  and  invites 
dialogue.  It  reveals  to  each  individual  the  biases  informing  their 
daily  evaluations  and  feedback  to  students.  And  it  takes  the  worry 
away  from  an  examination  re-mark  or  a discussion  with  parents.  El 
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For  a number  of  years,  the 
North  York  Board  has 
maintained  a strong  focus  on 
the  systematic  collection  of 
data  and  the  use  of  data  in 
planning  for  school  improve- 
ment. The  nature  of  the  data 
used  in  school  planning  and 
the  planning  processes  have 
evolved  over  the  years.  The 
Board  is  now  implementing 
a three-year  cycle  of  school 
self-reviews  in  which  School 
Improvement  Teams  conduct 
a comprehensive  analysis  of 


existing  data,  develop  an 
action  plan  to  address  at  least 
two  areas  for  improvement, 
implement  the  action  plan, 
and  monitor  the  success  of 
the  plan.  A critical  aspect  of 
the  review  process  is  that  data 
from  a wide  range  of  sources 
such  as  perception  surveys, 
system-wide  Benchmarks 
Tests,  teacher  assessments,  and 
demographic  information 
are  examined  and  analyzed 
within  the  context  of  the 
school  environment. 


Following  a system-wide  review  of  the 
Quality  Assurance  Program  in  1993-94, 
a decision  was  made  to  develop  a school 
self-review  process  which  would  enable  all 
schools  in  one  panel  to  conduct  a review 
in  a given  year.  In  addition  to  dealing 
with  the  issue  of  some  schools  not  being 
reviewed  for  several  years,  this  decision 
also  provides  for  a more  comprehensive 
system-wide  planning  initiative  and 
requires  all  schools  to  implement  a com- 
mon planning  and  improvement  model. 

School  Self-Reviews 

A primary  consideration  in  the  develop- 
ment of  the  self-review  model  was  to 
provide  a set  of  procedures  for  school 
staffs  that  would  enable  them  to  manage 
large  amounts  of  data,  to  use  those  data 
more  effectively  in  planning  for  school 
improvement,  and  to  monitor  their  suc- 
cess in  implementing  their  action  plans. 
Existing  sources  of  data  and  school  plan- 
ning requirements  were  streamlined  and 
incorporated  into  the  model  as  much 
as  possible. 

In  each  school,  the  self-review  is  con- 
ducted by  the  School  Improvement  Team 
consisting  of  a school  administrator,  teach- 
ers, parents,  and  possibly  students  in  sec- 
ondary schools.  This  team  works  very 
closely  with  the  School  Advisory  Council 
and  some  members  of  the  council  may 
also  be  on  the  School  Improvement  Team. 

Components  of  the  Self-Review  Cycle 

School  self-reviews  are  conducted  over 
a three-year  cycle. The  cycle  begins  with 
the  administration  of  perception  surveys 
to  parents,  teachers,  and  students  and  the 
completion  of  North  York  Benchmarks 
Tests  in  mathematics  and  literacy  (mathe- 
matics in  Grades  6,  8,  and  10;  literacy  in 
Grades  5,  7,  and  9).  During  the  first  year 
of  the  cycle,  the  following  tasks  are 
accomplished: 

• analysis  of  existing  data  to  produce  a 
description  of  the  current  status  of 
school  performance 

• identification  of  areas  of  success  and 
areas  where  improvement  is  required 


• in-depth  focus  on  the  areas  requiring 
improvement 

• description  of  one  area  of  outstanding 
performance 

• development  of  a School  Improvement 
Plan  for  areas  requiring  improvement, 
and  initiation  of  the  implementation  of 
the  plan 

• compilation  of  the  school  report 
During  the  second  and  third  years  of 

the  cycle,  the  school  implements  the 
school  improvement  plan  and  monitors 
the  success  in  achieving  the  specified  out- 
comes. Schools  continue  to  receive  student 
achievement  data  through  system-wide 
Benchmarks  Tests  and  provincial  assess- 
ments, and  they  also  collect  additional  data 
specifically  linked  to  the  improvement 
outcomes.  The  improvement  plan  is  modi- 
fied as  new  data  are  reviewed.  Following 
the  completion  of  the  three-year  cycle,  the 
school  will  begin  a second  self-review.  By 
the  year  2000,  all  North  York  schools  will 
have  completed  year  one  of  the  review 
process.  One  year  after  the  completion  of 
the  self  review,  an  external  audit  will  be 
conducted  in  a small  sample  of  schools  as 
a validation  check  on  the  process. 


The  procedures  used  to  complete 
each  of  the  components  of  the  self-review 
are  summarized  below.  Each  school 
receives  a manual  that  contains  the 
philosophical  foundation  for  the  review, 
a detailed  description  of  procedures, 
guiding  questions,  and  a number  of 
worksheets.  This  manual  is  also  provided 
in  electronic  format  which  assists  teams 
in  completing  the  worksheets,  summariz- 
ing their  responses,  and  completing  the 
final  report. 

Analysis  of  Existing  Data 

During  the  first  stage  of  the  self-review, 
the  School  Improvement  Team  draws 


upon  the  available  data,  information, 
knowledge,  and  experience  in  the  school 
to  describe  present  conditions  and  the 
current  status  of  the  school’s  performance. 
This  requires  a detailed  examination  of  the 
following: 

• existing  school  plan 

• perception  survey  results  for  students, 
teachers,  and  parents 

• student  achievement  data 

• implementation  of  system  priorities  and 
initiatives 

Worksheets  are  provided  for  each  of 
the  above  to  assist  the  team  to  bring 
together  data  from  a variety  of  sources, 
to  consider  the  data  collectively  rather 
than  as  isolated  components,  and  to  make 
connections  between  various  sources  of 


data  in  order  to  facilitate  the  identification 
of  school  trends  and  patterns,  and  areas  of 
success  and  challenge. 

For  example,  teams  are  asked  to 
examine  the  perception  survey  data  and 
identify  results  that  they  are  pleased  with 
or  concerned  about,  or  that  they  find 
inconsistent  or  confusing.  This  is  done 
through  comparing  teacher,  student,  and 
parent  perceptions  on  common  items; 
comparing  school  results  with  Board-wide 
results;  and  looking  at  changes  over  time. 

When  examining  achievement  data,  the 
team  is  expected  to  look  at  all  sources  of 
data  which  include  the  following:  on-going 
classroom  assessments,  the  Literacy 


In  each  school,  the  self-review  is  conducted  by  the  School 
Improvement  Team  consisting  of  a school  administrator,  teachers, 
parents,  and  possibly  students  in  secondary  schools. 
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...they  look  for  patterns  and  themes  across  data  sources  to  identify 


the  main  areas  where  the  school  has  shown  success  and  the  main 
areas  in  need  of  improvement. 


Assessment  Profile,  North  York  Benchmarks 
Tests  in  mathematics  and  literacy,  standard- 
ized tests,  provincial  assessments,  and 
program  reviews. 

Benchmarks  Test  results  are  provided 
to  schools  on  disk  with  a Filenraker  Pro 
package  that  enables  schools  to  compile 
and  print  a variety  of  reports  and  to  ana- 
lyze achievement  results  by  demographic 
categories.  Secondary  schools  have  a soft- 
ware program  called  Dataquest  for  analyz- 
ing information  from  central  data  bases 


that  enables  linkages  between  demographic 
data  and  achievement  results  in  secondary 
school  courses.  A number  of  pre-set 
reports  are  available  and  staff  members  can 
also  design  additional  analyses  to  produce 
tailored  reports. 

School  teams  may  also  seek  assistance 
from  Research  and  Assessment  Services 
in  interpreting  data  and  conducting 
additional  analyses. 


In-Depth  Focus  On  Areas 
Requiring  Improvement 

After  team  members  have  examined  and 
summarized  existing  data,  they  look  for 
patterns  and  themes  across  data  sources  to 
identify  the  main  areas  where  the  school 
has  shown  success  and  the  mam  areas  in 
need  of  improvement.  The  team  selects 
one  area  of  success  and  at  least  two  areas 
for  improvement  to  carry  forward  for  an 
in-depth  review  and  development.  During 
this  process,  team  members  will  also  iden- 
tify a number  of  areas  that  require  some 
attention  but  will  not  be  major  initiatives. 

The  team  conducts  a detailed  examina- 
tion of  the  school  to  determine  what 
is  working  well  and  what  is  not  working 
well  with  respect  to  the  areas  identified 
for  improvement.  This  component  of  the 
review  incorporates  the  following  general 
areas  of  focus  that  were  adapted  from 
the  National  Quality  Institute  guidelines 
for  evaluating  the  overall  effectiveness 
of  organizations: 

• school  direction  and  leadership 

• responding  to  and  meeting  the  needs 
of  students  (overall  student  focus) 

• planning  for  student  success 

• staff  optimization 

• process  analysis 

• partnering 

The  worksheets  for  this  section  provide 
a structure  for  the  in-depth  probing  of 
the  areas  selected  for  improvement.  Within 
each  of  these  areas,  the  team  responds  to 
a number  of  questions  regarding  the  level 
of  success. 

The  team  also  collects  additional  infor- 
mation in  order  to  answer  some  of  the 
questions  and  to  gain  insights  into  possible 


reasons  why  certain  procedures  are  not 
working  to  maximum  effectiveness.  Some 
of  this  information  will  be  in  existing 
school  records.  The  team  may  conduct 
individual  or  group  interviews  with  other 
staff  members,  parents,  and  students. 
Classroom  observations  are  another  source 
of  information. 

Description  of  an  Area 
of  Outstanding  Performance 

The  team  provides  a detailed  description 
of  one  area  of  successful  practice  within 
the  school.  This  description  is  also  struc- 
tured under  the  six  areas  of  focus  identi- 
fied above.  Within  each  of  these  areas  of 
focus,  the  team  describes  current  practices 
that  have  contributed  to  success  and/ or 
improvement,  provides  supporting  evi- 
dence, and  comments  on  future  directions 
for  the  initiative. 

School  Improvement  Plan 

The  main  focus  of  the  School  Improvement 
Plan  is  on  the  two  specific  areas  identified 
for  improvement,  but  it  includes  other 
areas  of  concern  that  may  be  identified. 
The  plan  must  also  deal  with  a wide  range 
of  school  initiatives  that  require  regular 
maintenance  and  an  occasional  infusion 
of  energy  and  enhancement.  In  the  devel- 
opment of  the  plan,  the  team  addresses 
the  following: 

Where  are  we  going? 

• learning  objectives,  outcome  statements, 
goals,  targets 

• suggested  indicators  or  measures 
of  success 

What  programs  or  strategies  will  we  use? 

• interventions,  expanded  opportunities, 
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teaching/learning  strategies,  innovative 
approaches,  processes  or  procedures 
What  help  do  n>e  need  to  get  us  there? 

• supporting  activities  which  are  neces- 
sary to  operationalize  the  plan 

• resources  (i.e.,  lists  of  school-based 
and/or  external  resource  requirements; 
plans  for  the  acquisition,  allocation, 
sharing  and  utilization  of  these 
resources) 

• new  knowledge  and/ or  skills  through 
professional  development  (recommen- 
dations for  PD  should  include  rationale, 
proposed  content,  participants  and  out- 
comes, budget) 

Who  are  the  people  who  will  help  get 
us  there? 

• an  indication  of  the  specific  people 
(i.e.,  teachers,  administrators,  support 
staff,  central  Board  staff)  who  are 
responsible  for  carrying  out  each  of  the 
tasks  laid  out  in  the  plan 

When  will  we  get  there? 

• a fixed  timeline  that  schedules  and 
sequences  all  the  tasks  and  activities 
covered  in  the  plan 

• expectations  for  short-  and  long-term 
goals 

How  will  we  know  that  we  are  there? 

• monitoring  procedures  for  data 
collection  at  specified  intervals 

School  Report 

The  worksheets  completed  during  the 
conduct  of  the  review  constitute  the 
school  report.  The  main  body  of  the  report 
consists  of  the  description  of  the  areas 
identified  for  improvement,  the  description 
of  the  area  of  outstanding  performance, 
and  the  School  Improvement  Plan.  The 
worksheets  used  for  analyzing  and  summa- 
rizing data,  and  the  worksheets  on  the  m- 
depth  review  of  the  areas  for  improvement 
provide  supporting  documentation. 

Board  Report 

A report  for  the  Board  that  includes  the 
Board-wide  results  for  the  perception 
surveys  and  a summary  of  some  of  the 
content  of  the  school  reports  is  produced. 
The  areas  of  outstanding  performance  are 


highlighted  so  that  these  can  be  shared 
across  all  schools  in  the  Board.  The  areas 
identified  for  improvement  are  reviewed 
to  identify  common  themes  that  have 
been  identified  by  several  schools.  If  a 
large  number  of  schools  identify  particular 
needs,  it  will  be  important  to  provide  sup- 
port from  central  Board  departments  to 
assist  schools  in  dealing  with  these  needs. 
The  Board  report  provides  valuable  infor- 
mation for  strategic  planning. 

Communication 

Each  school  develops  a communication 
plan  to  inform  all  of  its  stakeholders  of  the 
key  findings  of  the  review.  Different 
aspects  of  the  review  are  highlighted  for 
different  audiences.  The  North  York 
School  Profiles  provide  a mechanism  for 
informing  parents  about  the  key  compo- 
nents of  the  school  improvement  plan  and 
about  annual  progress  toward  achieving 
the  goals  of  the  plan. 

The  Board  also  produces  a number 
of  communications  about  the  process  and 
about  the  key  findings  for  the  review  at 
each  panel. 

In-Service  for 

School  Improvement  Teams 

At  the  beginning  of  the  process,  School 
Improvement  Teams  receive  in-service  on 
the  conduct  of  self  reviews  which  covers 
the  following: 

• relating  of  quality  principles  to  school 
planning  and  improvement 

• understanding  and  interpreting  assess- 
ment data 

• linking  achievement  data  to  school 
improvement 

• communicating  results 

• following  the  specific  procedures  devel- 
oped for  the  self  reviews 

• completing  the  worksheets  and  using 
the  electronic  version  of  the  manual 

Summary 

The  three-year  cycle  of  school  self  reviews 
being  implemented  in  North  York  schools 
was  developed  to  provide  a mechanism  to 
incorporate  the  management  of  large 


amounts  of  data  into  the  school  planning 
process.  The  self  review  was  designed  to 
facilitate  the  integration  of  a number  of 
initiatives  and  action  plans  that  schools 
have  been  required  to  deal  with  on  an 
annual  basis.  The  critical  features  of  this 
review  process  are  school  planning  for 
improvement  based  on  data  and  evidence, 
and  the  monitoring  of  progress  toward 
achievement  of  the  goals  identified  in  the 
improvement  plan. Through  the  conduct 
of  the  self  review  school  staffs  became 
competent  in  interpretation  and  use  of 
data  and  develop  skills  that  enable  them 
to  evaluate  their  own  work  and  respond 
more  effectively  to  changing  needs. 
Schools  conduct  a comprehensive  internal 
review  and  develop  a detailed  School 
Improvement  Plan  during  the  first  year  of 
the  cycle.  During  the  second  and  third 
years  of  the  cycle,  schools  implement  then- 
improvement  plans  and  monitor  progress 
toward  achieving  the  objectives  of  the 
plan.  A new  cycle  is  begun  after  the  com- 
pletion of  the  three-year  cycle.  B 
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Assessment 
Systems,  Outcomes, 
and  Benefits 

John  S.  Clipsham 

Niagara  South  Board  of  Education 

Accountability  requires  developing 
and  communicating  clear  statements 
of  expected  outcomes;  these  serve  as 
standards  for  assessment,  and  involve  staff, 
students,  and  community  in  partnerships 
which  should  enhance  success. 

Accountability  also  requires  appropriate 
decisions  about  measures  to  be  used  to 
determine  the  extent  to  which  desired 
outcomes  are  being  achieved.  System-wide 
assessments  can  only  address  a small 
portion  of  the  curriculum  at  a limited 
number  of  points  in  time  and  usually  with 
only  a limited  number  of  students;  the 
aim  is  to  use  a variety  of  “dipsticks.” 


In  Niagara  South,  assessments  based 
on  Board  curricula  are  provided  for 
mathematics,  language  arts,  problem  solv- 
ing, and  computer  skills.  In  mathematics 
and  language  arts,  all  students  are  assessed 
m selected  grades;  in  the  others,  samples 
of  students  are  assessed.  The  mathematics 
assessments  are  paper-and-pencil  tests 
administered  at  particular  times  of  year; 
the  language  arts  measures  are  perfor- 
mance tests  or  observations  covering  key 
outcomes  in  listening,  speaking,  reading 
and  writing,  and  are  administered  where 
they  fit  in  instruction  over  an  entire 
course.  To  address  accountability  for 
achievement  of  outcomes  in  other  parts 
of  the  curriculum,  we  have  periodically 
developed  and  administered  performance 
measures  to  selected  students  (provincial 
reviews  also  serve  this  function). We  have 
also  implemented  a policy  that  requires 
principals  and  department  heads  to  review 


files  of  assessment  instruments  and  proce- 
dures regularly  to  ensure  all  courses  assess 
students  consistently  with  the  program 
information. 

For  secondary  schools,  we  have  imple- 
mented “consistency  agreements”  which 
describe  the  content  and  the  types  of 
items  to  be  used  for  the  25  to  40  percent 
of  the  course  mark  which  is  the  final 
summative  assessment.  Data  from  centrally 
monitored  evaluation  files  and  final  assess- 
ments help  the  Board  to  validate  that  there 
is  consistency  in  the  measures  used  for 
evaluating  student  achievement  from  class 
to  class  and  school  to  school.  We  have  used 
nationally  normed  tests  to  assess  reading, 
academic  potential,  and,  for  several  years, 
fitness  (costs  of  processing  the  data  and 
questions  about  their  validity  led  us  to 
stop  analyzing  the  data  centrally,  though 
many  schools  continue  to  participate  in 
fitness  assessments). 
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Types  of  data  less  directly  related  to  the 
assessment  of  student  achievement  are  also 
needed  when  a board  seeks  to  be  account- 
able for  assessing  Strategic  Plan  outcomes, 
documenting  the  system’s  input  to  support 
effective  classroom  delivery,  documenting 
the  teaching/learning  process,  and  the  type 
and  extent  of  implementation  of  Board  or 
Ministry  directions.  Evaluation  of  Strategic 
Plan  outcomes,  for  example,  involved  us  in 
many  types  of  data  collection  and  analysis. 
Surveys  of  current  students,  graduate  and 
dropout  school  leavers,  parents,  employers, 
and  staff  members  asked  for  stakeholder 
perceptions  about  how  well  we  were 
doing  with  our  mission  statement  and 
principles  and  how  well  we  were  attaining 
the  outcome  indicators  of  success  listed  in 
the  Plan.  To  aid  us  in  interpreting  some 
aspects  of  the  questionnaire  data,  we  held  a 
series  of  forums  or  focus  groups  with 
stakeholder  groups;  these  also  included 
opportunities  for  input  from  university 
and  community  college  teachers,  social 
agency  partners,  and  local  chambers  of 
commerce.  Other  Strategic  Plan  indicators 
included:  per-student  expenditures  and 
evidence  of  cost-saving  or  entrepreneur- 
ship  in  obtaining  resources  or  utilizing 
volunteer  or  partner  services;  participation 
in  staff  development  activities;  employ- 
ment equity  statistics;  enrolment  in  pro- 
grams such  as  technological  studies; 
dropout  and  graduation  and  course 
completion  rates  and  profiles  (e.g.,  gradu- 
ate destinations  and  distributions  of  marks 
into  As,  B’s,  etc.);  student  and  employee 
attendance  patterns. 

When  particular  programs  are  being 
evaluated,  it  is  common  to  gather  student 
attitude  data  and  teacher  perceptions, 
but  a variety  of  other  measures  such  as 
classroom  observation,  parent  opinions, 
and  surveys  of  resource  utilization  may 
be  useful;  these  provide  a context  for 
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discussing  the  student  achievement  data. 

In  addition,  the  Board  maintains  a variety 
of  data  concerning  such  issues  as  class 
size,  course  enrolments,  qualifications  and 
experience  of  staff,  enrolment  in  special 
education  programs,  promotion  and 
advancement  statistics,  needed  upgrades 
to  equipment  and  facilities  — all  data 
which  are  sometimes  valuable  for  identify- 
ing trends  or  accountability. 

Inspection  of  the  variety  of  data  already 
listed  will  indicate  the  variety  of  uses  to 
which  accountability  data  might  be  put. 
Nonetheless,  it  must  be  emphasized  that 
being  accountable  has  significant  costs, 
in  terms  of  personnel  time  and  expertise 
to  collect  and  analyze  data,  as  well  as 
material  resources.  In  today’s  climate,  some 
of  those  costs  seem  warranted  to  ensure 
that  school  boards  can  be  accountable  to 
their  stakeholders;  on  the  other  hand, 
all  costs  in  education  are  under  scrutiny 
and  these  costs  plus  the  time  required  of 
students  and  front-line  staff  in  providing 
the  data  may  be  considered  resources 
taken  away  from  classroom  instruction. 
That  is  why  anticipated  benefit  for 
students  and,  through  them,  for  the  com- 
munity must  be  emphasized  in  planning 
accountability  processes.  The  Board- wide 
monitoring  of  student  achievement  pro- 
vides some  illustrations: 

• Different  achievement  measures  serve 
different  purposes  and  may  have  differ- 
ent teacher-users.  For  example,  our 
mathematics  pre-tests  assess  the  extent 
to  which  students  have  retained  into 
the  fall  of  Grade  4 or  7 skills  learned 
over  the  previous  three  years;  Grade  4 
and  7 teachers  can  uke  the  individual 
student  profiles  to  place  them  on  the 
various  skill  continua  and  to  plan  fur- 
ther review  or  remedial  opportunities. 
Teachers  of  previous  grades  can  use 
data  about  the  school’s  areas  of  strength 


and  weakness  to  target  parts  of  their 
programs  for  improvement  efforts. 

• For  each  system-wide  achievement 
measure,  teacher-users  of  the  data 
are  given  support,  with  profiles  of 
individual  student  and  class  group  data. 
These  data  are  both  norm  and  criteri- 
on-referenced, with  guidelines  for 
interpretation  and  suggestions  for 
classroom  follow-up. 

• Principals  receive  annual  updates  to 
school  profiles  which  include  summary 
statistics  for  each  achievement  measure 
for  the  previous  five  years  as  well  as 
academic  aptitude  averages  which  show 
how  the  school  compares  with  its  past 
performance,  the  system  averages,  and 
the  level  expected  reflecting  student 
cognitive  abilities.  These  profiles  may 
be  used  to  select  subject-area  priorities 
for  the  school  improvement  plan,  to 
determine  whether  there  are  gender 
differences  in  achievement  or  whether 
there  are  satisfactory  numbers  of 
students  in  the  high  and  low  achieve- 
ment sectors  of  the  school  population. 

• Superintendents  use  system  achieve- 
ment data  in  monitoring  the  schools 
for  which  they  are  responsible  and 
helping  principals  to  set  improvement 
targets. 

• Curriculum  Services  examine  the 
system  achievement  data  for  needed 
changes  in  program  documents  or 
possible  topics  for  staff  development 
activities,  while  Special  Services  consid- 
er the  implications  of  the  data  tor 
identifying  exceptional  students  and 
tracking  the  appropriateness  of  their 
programs. 

• Exploration  of  uses  of  student 
achievement  data  by  parents  is  ongoing. 
School  Advisory  Councils  may  be 
involved  in  helping  set  program 
improvement  priorities,  and  individual 
parents  may  partner  with  teachers  in 
responding  to  the  individual  student’s 
profile  of  performance. 

In  Niagara  South,  there  is  no  direct 

evidence  of  an  achievement  decline  in 

recent  years.  Flowever,  data  from  parents, 


Principals  receive  annual  updates  to  school  profiles  which 
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postsecondary  teachers,  and  employers 
obtained  while  evaluating  Strategic  Plan 
implementation  all  showed  great  concern 
for  student  performance  on  traditional 
basic  skills.  Accordingly,  the  Board  feels 
justified  in  continuing  to  monitor  and 
seek  improvement  in  basic  literacy  and 
numeracy  skills.  Moreover,  our  own  data 
suggest  achievement  in  reading  falls  some- 
what below  our  expected  and  desired  lev- 
els; accordingly,  the  Board  has  undertaken 
a major  initiative  to  support  teachers  in 
teaching  reading  strategies.  In  addition, 
secondary  teachers  have  reported  con- 
cerns about  student  attendance  impacting 
achievement;  examination  of  class  atten- 
dance patterns  tends  to  validate  some  of 
these  concerns,  and,  once  again  the  Board 
is  using  this  information  as  the  basis  of 
an  initiative  to  seek  to  improve  student 
attendance,  in  support  of  improved 
student  achievement. 

On  the  nationally-normed  Gates- 
MacGinitie  Reading  Test,  the  Boards  stan- 
dard scores  over  the  last  10-14  years  had 
been  tending  to  decline  slightly  at  Grade  3 
and  8 but  not  at  Grade  1 1 where  a slight 
increase  in  vocabulary  scores  balanced  a 
slight  decrease  in  comprehension  scores. 
When  the  recently  revised  and  norrned 
measures  were  administered  in  1996  at 
Grade  8 and  1 1 , the  comprehension  scores 
jumped  back  to  their  former  levels. 

On  measures  referenced  to  the  mathe- 
matics curriculum,  the  trend  over  the  last 
decade  has  been  for  increases  in  achieve- 
ment — mostly  in  the  early  years  of  the 
decade  at  Grade  4 and  7 and  in  the  last 
three  years  for  General  and  Advanced  stu- 
dents at  Grade  10.  This  improvement  at 
Grade  1 0 occurred  during  the  period  of 
the  Transition  Years  initiatives  and  during 
a time  when  higher  proportions  of  stu- 
dents chose  these  two  higher  levels  of  dif- 
ficulty; that  is,  lower  scores  rather  than 


increases  might  well  have  been  predicted. 
Increases  were  most  pronounced  in  areas 
of  the  curriculum  which  have  been  added 
in  recent  years. 

On  measures  referenced  to  the  Board’s 
language  arts  curriculum  administered 
since  1988,  there  has  been  no  significant 
decline  in  success  in  reading,  writing, 
listening,  or  speaking  in  Grade  3,  6, 
or  8,  and  in  most  cases  there  was  an 
improvement  in  the  first  few  years  of 
administration  and  no  change  in  the  last 
five  years.  On  a tour-part  assessment  of 
problem-solving  at  Grade  6,  the  average 
scores  on  skills  ranging  from  clarifying 
the  problem  to  reporting/ explaining  the 
findings  increased  by  10  percent  over 
the  last  decade,  mostly  in  the  early  years. 
This  assessment  covers  areas  such  as 
critical  reading,  library  research,  decision 
making,  social  studies,  and  science  as  well 
as  mathematics. 

Over  the  last  seven  years,  it  appears 
that  secondary  school  dropout  rates  first 
declined,  and  then  stabilized;  however, 
even  during  the  period  of  stable  rates, 
students  appear  to  be  staying  in  secondary 
school  longer  (typically  at  least  four 
years)  and  more  seem  to  be  re-enrolled 
or  continuing  with  some  form  of  educa- 
tion when  checked  several  months  after 
dropping  out. 

Longest  Continuing  Measure 

In  Niagara  South,  the  longest  continuing 
measure  is  the  Grade  8 Mathematics 
Fundamentals  Test  which  was  first  adminis- 
tered in  1977-78.  Over  this  period,  the 
average  score  (marks  only  awarded  for  con- 
structing the  correct  answer)  rose  from 
62.2  percent  to  a peak  of  70.9  percent  in 
1993,  and  have  held  at  70  percent  the  last 
three  years.  During  that  period,  the  test  has 
never  been  “high  stakes”  in  terms  of 
counting  as  part  of  the  mathematics  mark 


or  in  sorting  students  for  Grade  9 programs 
(though  assessments  showed  a high  correla- 
tion with  Grade  9 marks  and  some  Grade 
9 teachers  did  use  the  results  in  counselling 
students) . For  several  years,  results  on  this 
test  were  prominent  in  Board  and  school 
improvement  discussions,  but  in  recent 
years,  attention  given  to  the  test  has  been 
at  the  discretion  of  individual  principals 
and  teachers.  Certainly,  the  test  is  biased 
toward  conventional  views  of  mathematics 
fundamentals  and  does  not  well  reflect  all 
of  the  mathematics  strands  in  the  Common 
Curriculum.  A key  point  here  is  that 
improvement  has  been  demonstrated 
on  conventional  basics,  during  a period 
in  which: 

• the  scope  of  the  mathematics  curricu- 
lum has  expanded  substantially,  with 
“new”  topics  often  becoming  a focus 
of  teacher  attention 

• the  number  of  exceptional  students 
integrated  in  regular  classes  and  there- 
fore included  in  the  testing  and  tend- 
ing to  lower  scores  has  increased 

• fewer  elementary  school  students  were 
retained  (spending  an  extra  year  in 
elementary  school),  resulting  in  a 
decrease  in  the  average  age  of  the 
Grade  8 students 

• there  have  been  significant  additions 
elsewhere  in  the  curriculum  (e.g., 
learning  new  technologies  such  as 
computers  and  calculators,  new  health 
issues,  career  education,  and  increased 
focus  on  science  and  technology) 

All  of  these  other  trends  might  well 
have  contributed  to  a decline  in  scores 
on  this  test,  but  did  not  produce  an 
overall  decline  on  this  or  any  of  the  other 
measures  we  have  used. 

Because  change  in  education  is 
continuous,  it  is  likely  that  any  given 
indicator  of  effectiveness  will  not  be  a 
constant  focus  of  improvement  efforts. 

The  encouraging  finding  from  most  of 
these  indicators  is  that  improvements 
are  possible  during  a period  of  sustained 
attention,  and  that  these  improvements 
tend  to  be  maintained  even  when  priori- 
ties shift  elsewhere,  d 
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Improvement  is  the  key  focus  or  outcome  of  accountability. 

Accountability  systems  are  the  most  effective  when  they  are 
implemented  at  the  local  school  level,  issue-specific,  congruent 
with  the  school’s  mission/goals/ planning  process,  and  designed  to 
improve  teaching  and  learning,  rather  than  giving  priority  to 
narrowly  defined  outcomes. 

This  article  summarizes  one  school’s  experience  with  integrat- 
ing planning  for  improvement  and  accountability;  the  school’s 
improvement  efforts  are  discussed  within  the  context  of  basic 
accountability  approaches. 

SCHOOL  SUCCESS  PLANNING 
AND  ACCOUNTABILITY  IN  PEEL 

For  the  past  five  years,  the  Peel  Board  of  Education  has  been 
involved  in  a School  Success  Planning  Project.  The  project  was 
designed  to  encourage  continuous  improvement  and  increased 
student  success  through  a school-based  planning  process  based  on 
the  collection  of  data  to  assess  relevant  outcomes.  This  approach 
has  been  implemented  in  Peel  at  the  system  level,  the  school 
level,  and  the  student  level. 

As  part  of  this  project,  every  Peel  school  has  been  expected  to 
participate  in  a planning  and  assessment  process  and  to  develop  a 
multi-year  school  success  plan.  The  general  objectives  of  a school 


success  plan  are:  (1)  to  increase  the  achievement  of  learning 
outcomes  for  all  students;  (2)  to  improve  the  school’s  program, 
instruction,  and  services;  and  (3)  to  meet  local  community 
accountability  needs.  Each  school’s  plan  should  describe  its  goals 
for  continuous  improvement,  and  the  activities  in  which  it  will 
engage  to  achieve  this  success. 

The  plan  is  to  be  developed  by  a school  success  team,  made  up 
of  the  principal  and/ or  vice-principal,  staff  members,  and  some- 
times parents  and/or  students.  The  team  works  through  a process 
that  includes:  (1)  evaluating  all  aspects  of  the  school’s  learning 
environment;  (2)  developing  a plan  to  meet  the  school’s  needs; 

(3)  generating  strategies  to  implement  the  plan;  (4)  evaluating 
and  reporting  on  the  outcomes  of  the  school  plan;  and  (5) 
ensuring  renewal  of  the  school  plan  over  subsequent  years. 
Over  the  past  five  years.  Staff  Services  and  the 
Assessment  and  Evaluation  Department  have  provided 
inservice  to  school  success  teams  in  such  areas  as  readi- 
ness for  change;  school  culture/ climate;  vision/ mis- 
sion/values; correlates  of  effective  schools;  gathering 
and  interpreting  data;  building  a school  profile  and 
portrait;  developing  and  implementing  a school 
action  plan;  and  evaluating  the  success  of  the 
school  plan. 


CASE  STUDY:  THE  STORY 
OF  McHUGH  PUBLIC  SCHOOL 

McHugh  Public  School,  Brampton,  Ontario,  built  in  1923,  has  a 
current  enrollment  of  350  students  in  kindergarten  to  Grade  5, 
with  three  special  education  classes. The  school  draws  on  a largely 
Canadian-born  and  very  mixed  socio-economic  population  and 
scores  within  the  average  range  on  tests  of  learning  aptitude  and 
reading,  spelling,  and  mathematics  achievement. 

School  success  planning  at  McHugh  was  undertaken  with  an 
understanding  that  effecting  change  takes  time,  must  be 
approached  in  an  organized  manner,  involves  all  stakeholders,  is 
guided  by  a solid  research  base,  and  rests  on  both  formal  and 
informal  data  collection. 

Everyone  involved  with  McHugh  — students,  parents,  teachers, 
administration,  and  other  staif — has  participated  in  the  school 
success  effort  at  one  time  or  another  over  the  past  few  years.  The 
staff  has  had  to  develop  new  skills  and  insights  in  order  to  orches- 
trate ownership  of  the  process  by  the  school  community, 
develop /implement/evaluate  an  action  plan,  and  report  on  the 
results.  They  have  had  to  keep  abreast  of  the  educational  reform 
literature,  exercise  skills  in  group  dynamics,  understand  the  change 
process,  commit  to  a vision,  maintain  an  optimistic  attitude  about 
the  future,  and  sometimes  engage  in  risk-taking  behaviors. 

Before  the  actual  journey  began,  a year  of  learning  (1990- 
1991)  was  devoted  to  gathering  data  and  information  on 
McHugh’s  history  and  the  characteristics  and  perceptions  of  its 
students,  staff,  parents,  and  community.  During  this  year,  the 
school  developed  its  mission  statement:  “We  will  improve  the 
standards  of  teaching  and  learning  and  the  quality  of  life  for  McHugh’s 
students,  parents  and  teachers.  ” This  provided  the  foundation  for 
future  planning  at  McHugh. 

Year  1 (1991-92):  Year  of  Transition 

The  data  collected  in  1990-1991  helped  the  school  identify  three 
strands  as  a focus  for  the  first  year:  social  skills/self-esteem  building, 
behavior  management,  and  active  learning.  Led  by  three  subcom- 
mittees, a number  of  initiatives  were  implemented  — for  example, 
a series  of  activities  to  enhance  students’  self-esteem,  a set  of  school 
rules  for  parents  to  review  with  their  children,  Peers  who  Agree 
to  Listen  and  Solve  Problems  program  (PALS),  and  classroom 
management  techniques  and  social  skills-building  activities. 

Each  initiative  was  evaluated  by  its  subcommittee,  and  the 
results  were  shared  with  staff  and  administration.  Classroom 
newsletters  reported  on  the  progress  of  the  initiatives  and  offered 
suggestions  to  parents  about  activities  that  could  assist  their  child. 
The  home-school  link  was  further  strengthened  by  continuing 
weekly  parent  drop-ms  and  by  having  the  school  administrators 
and  two  staff  members  attend  PTA  meetings.  Data  collection 
continued  to  be  a focus  for  planning  and  evaluation.  It  was  decid- 
ed to  accumulate  information  on  playground  incidents  and  to 
re-adnnmster  the  Peel  School  Success  Parent  and  Teacher  surveys. 


Year  2 (1992-1993):  Year  of  Implementation 

In  Year  2,  McHugh  continued  its  focus  on  implementing  and 
evaluating  programs,  and  on  communicating/ reporting  to  parents. 
All  initiatives  were  reviewed  to  ensure  that  they  were  consistent 
with  a developing  school  culture  centered  on  openness,  collabo- 
ration, and  communication.  The  school  success  team  attended  five 
inservice  modules  offered  by  Staff  Services  and  the  Assessment 
and  Evaluation  Department,  designed  to  broaden  teams’  under- 
standing of  data-based  school  planning. 

Every  McHugh  teacher  was  expected  to  join  one  of  the 
school  success  planning  committees,  which  had  expanded  to 
include  academic  support  staff  and  parent  representatives.  A 
Parents  and  Teachers  Talking  Together  program  (PATTT)  was 
implemented  and  extended  to  include  all  parents.  Efforts  to  build 
self-esteem  continued  with  the  introduction  of  student-run 
assemblies  to  recognize  children’s  accomplishments. 

Year  3 (1993-1994):  Year  of  Growth 

By  Year  3,  McHugh  had  markedly  improved  its  capability  for 
refining  existing  programs,  initiating  new  ones,  and  communicat- 
ing/reporting  to  parents.  A Skills  for  Growing  curriculum, 
developed  by  Lions  Club  International  and  Quest  International, 
was  introduced  on  a school-wide  basis.  This  program  was 
designed  to  bring  together  parents,  educators,  and  community 
members  to  teach  McHugh  students  important  life  skills. 

The  focus  on  behavior  management  was  maintained  in  Year  3, 
with  a continuing  emphasis  on  school  rules.  This  effort  became 
more  collaborative  in  nature  and  began  to  incorporate  input 
from  students.  Parent-teacher  dialogue  was  strongly  encouraged, 
and  information-sharing  sessions  were  held  in  conjunction  with 
curriculum  nights. 

McHugh  teachers  continued  to  emphasize  collaborative 
learning  as  a way  of  enhancing  student  achievement  outcomes. 
Portfolio  assessment  was  implemented  in  order  to  track  and 
monitor  student  progress  over  the  school  year  and  across  grades 
and  teachers  were  organized  into  small  within-  and  across- 
grade  groupings  that  met  on  a bi-monthly  basis  to  discuss 
professional  issues  and  students’  progress. 

An  instructional  resources  team  was  also  established 
to  assess  the  social,  emotional,  behavioral,  and  academic 
development  of  particular  students.  When  a school- wide 
testing  program  indicated  that  some  McHugh  students 
were  having  reading  difficulties,  the  team  studied 
the  current  research  on  phonemic  awareness  in 
order  to  develop  an  early  intervention  plan  for 
the  following  year. 

A breakfast  program  was  established  at  the 
school  to  ensure  that  students  started  the 
instructional  day  with  adequate  nourishment. 

Community  partnerships  were  formed  to 
maintain  the  program  for  its  start-up  year. 
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Year  4 (1994-1995):  Year  of  Moving  On 

In  Year  4,  an  Information  Technologies  Committee  was  formed  to 
reinforce  a computers-across-the-curriculum  initiative.  This  com- 
mittee (which  included  parent  and  community  representatives), 
established  a computer  lab  to  ensure  that  all  students  had  access 
to  computers  and  were  being  taught  a variety  of  computer  skills. 
Teachers  were  also  expected  to  upgrade  and  maintain  their 
computer  skills.  The  implementation  of  McHugh’s  computer  plan 
was  greatly  facilitated  by  a gift  of  $50,000  in  computers  and  relat- 
ed equipment  from  a private  foundation. 

In  addition  to  the  emphasis  on  information  technology, 
McHugh  implemented  or  continued  to  develop  the  following 

programs:  (1)  a phonemic  awareness 
initiative  to  enhance  reading  skills  in 
K-2  students;  (2)  the  Skills  for 
Growing  program  and  the  Parents 
and  Teachers  Talking  Together  pro- 
gram; (3)  introduction  of  anger 
management  workshops  and  a 
peer  mediation  program  to  ensure 
a safe,  secure  environment  and 
promote  personal  responsibility; 


Everyone 
involved  with 
McHugh  has 
participated 
in  the  school 
success  effort 
at  one  time 
or  another  over  the 
past  few  years. 


and  (4)  additional  support  for 
teachers  preparing  personal 
Student  Success  Plans  (i.e., 
consultation,  classroom 
observations,  group  support). 
During  Year  4,  McHugh 
developed  its  school  portrait, 
an  important  component  of 
Peel’s  planning  and 

accountability  process.  The  school  portrait  is  intended  to  provide  a 
comprehensive,  data-based  picture  of  a school:  it  documents  the 
school’s  history,  programs,  facilities,  resources,  organization,  and 
student  demographics;  it  reports  on  the  school  climate  and  pre- 
sents summaries  of  student  achievement  measures;  it  describes  how 
various  groups  of  stakeholders  perceive  the  school;  and  it  details 
the  school’s  action  plan  and  how  it  will  be  evaluated. 


ment,  teach  social  skills,  and  provide  professional  development/ 
support  for  kindergarten  teachers. 

The  school’s  reading  improvement  initiative  was  extended 
from  K-2  to  K-3,  and  the  program  was  evaluated  in  order 
to  identify  areas  for  planning  and  improvement.  The  school 
administrators  instituted  annual  personal  success  reviews  with 
individual  staff,  during  which  classroom  programs  and  student 
progress  were  discussed. 

DISCUSSION 

School  success  planning  and  accountability  have  become  a part 
of  the  day-to-day  life  at  McHugh.  The  staff  do  not  view  these 
efforts  as  additional  work,  but  as  part  of  a consistent  effort  toward 
improving  student  success. 

The  journey  has  taken  them  more  than  five  years.  It  has 
involved  all  relevant  stakeholders  in  a collaborative  planning  and 
reporting  process,  and  it  has  engendered  a common  vision  for 
the  school.  McHugh  staff  agree  that,  even  with  a clear  destination 
in  mind,  they  could  not  have  made  the  journey  alone.  Parents, 
business  partners,  and  the  community  have  shown  strong  support. 
Involvement,  honesty,  planning,  and  open  communication  have 
been  the  cornerstones  of  this  success. 

Others  have  recognized  the  uniqueness  of  the  school’s  efforts. 
In  1994,  McHugh  received  the  Reader’s  Digest  Leadership  in 
Education  Award  for  its  work  in  school  success  planning.  The 
co-chairs  of  the  school  success  team  have  also  been  invited  to 
present  their  planning  and  accountability  model  to  schools  in 
New  Zealand.  Through  collaborative  planning  and  an  awareness 
of  accountability  issues,  McHugh  staff  have  begun  the  process  of 
living  their  mission. 

The  McHugh  Public  School  case  study  demonstrates  that 
real  school  improvement  depends  on  a well  articulated  planning 
model  that  includes  the  whole  school  staff  in  its  development, 
implementation,  and  evaluation;  on  credible  data  regarding 
many  aspects  of  school  functioning  and  student  learning; 
and  on  an  adequate  system  of  communicating  the  planning 
process  and  action  plan  to  various  stakeholder  groups  in  the 
community.  El 


Year  5 (1995-1996):  Year  of  Refinement 

Year  5 of  the  school  success  planning  and  accountability  process 
found  McHugh  facing  such  challenges  as  the  reduction  of 
professional  development  days,  increased  enrollment,  new  staff 
members,  and  accommodation  concerns.  As  a result  of  collabora- 
tive planning  (involving  staff,  parents,  and  students),  the  school 
decided  to  concentrate  in  Year  5 on  refining,  refocusing,  and 
strengthening  existing  priorities. 

Year  5 priorities  — along  with  maintaining  previous  initiatives 
— included  implementing  a school-wide  anger  management 
and  empathy  training  program,  and  reinstating  the  Kindergarten 
Intervention  Program  (KIP),  designed  to  increase  parent  involve  - 


References 

Kogan,  M.  (1986).  Educational  accountability : An  analytic  overview.  London: 
Hutchinson. 

Lundgren,  U.P.  (1990).  Educational  policy  making,  decentralization  and 
evaluation.  In  M.  Granheim,  M.  Kogan,  & U.P.  Lundgren  (Eds.),  Evaluation 
into  the  national  decentralized  education  system  (pp.  23-41).  London:  Jessica 
Kingly. 

Maxwell, T.W.  (1996).  Accountability:  The  case  of  accreditation  of  British 
Columbia’s  public  schools.  Canadian  Journal  of  Education,  21,  18-34. 

Taylor,  A.,  & Hill,  F.  (1993).  Quality  management  in  education.  Quality 
Assurance  in  Education , 1,21-28. 


School  Improvement  i 

through  Self-Referenced 

Student  Assessment 


Connie  Kidd 

Hamilton  Board  of  Education 

Schools  are  expected  to  have  visible 
plans  for  improvement,  and  to  demon- 
strate improvement  in  students’  results. 
School  improvement  is,  in  fact,  the 
improvement  ot  student  results  in  impor- 
tant knowledge,  skills,  and  behaviour. 

School  plans  are  maps  for  improvement 
which  include: 

• areas  of  focus  identified  through  needs 
assessment 

• specific  outcomes  with  indicators  for 
assessing  success 

• school  and  classroom  strategies  and 
implementation  plans 
• review  of  evidence  for  effectiveness 
(student  results)  and  efficiency 
(implementation  and  use  of  resources) 

• identification  of  next  steps  — new 
focuses,  outcomes,  indicators,  and 
strategies 

Good  evidence  for  a plan’s  effectiveness 
is  critical  to  fulfilling  the  purpose  of  the 
plan  — bringing  about  demonstrable 
improvement  in  student  results.  It  is  also 
important  for  maintaining  the  motivation 
of  students  and  teachers  and  the  involve- 
ment of  parents.  Results  from  student 
assessments  may  be: 

• group-referenced  (comparing  a student’s 


performance  to  that  of  others 
of  his  or  her  age,  grade,  or  class,  e.g., 
CAT/2,  CTBS,  providing  median 
scores  on  report  cards) 

• standards-referenced  (comparing  a stu- 
dent’s performance  to  predetermined 
expectations  for  a grade  such  as  the 
provincial  standards  in  language  and 
mathematics,  e.g.,  provincial  review 
results,  provincial  report  cards) 

• self-referenced  (comparing  a student’s 
performance  to  his  or  her  prior 
performance) 

Some  evidence  about  student  perfor- 
mance is  periodically  available  to  schools 
in  summaries  of  student  performance  from 
group-referenced  or  standards-referenced 
assessments.  These  are  valuable  indicators 
of  students’  achievement  in  certain  grades 
at  certain  points  in  time,  and  provide  gen- 
eral direction  for  improvement.  However, 
schools  also  need  to  collect  their  own 
performance  data  with  reference  to  the 
specific  needs  of  their  students,  focuses  of 
their  school  plan  programs  in  their  class- 
rooms, and  community  expectations.  Self- 
referenced  assessment  is  particularly  useful 
in  schools  as  it  reflects  the  teaching-learn- 
ing cycle  of  classrooms  and  is  relevant  to 
what  administrators,  teachers,  students,  and 
parents  want  to  know.  Ultimately,  whether 
students  compare  well  to  a group  or  a 


standard  or  not,  teachers  assess  to  identify 
what  the  students  know  and  can  do  and 
what  comes  next,  plan  and  deliver  an 
appropriate  program  of  instruction,  and 
reassess  to  evaluate  progress  and  report 
to  students  and  parents. 

Relevant  classroom  assessment,  which 
is  agreed  upon  and  shared  across  classes 
and  grades,  can  be  summarized  for  classes 
and  whole  schools  to  demonstrate  accom- 
plishments and  identify  next  steps  for 
each  student  and  the  school  as  a whole. 
Shared  classroom  assessment  data  can  be 
used  to  monitor  progress  toward  academic 
standards  and  other  classroom  and  school 
goals,  to  adjust  school  and  classroom 
strategies  in  progress,  and  to  provide 
frequent  feedback  to  students,  parents 
and  the  community. 

“If  the  purpose  of  assessment  and 
evaluation  in  schools  includes  improving, 
not  merely  monitoring  student  perfor- 
mance, the  clients  (particularly  students) 
need  ongoing  and  detailed  feedback  about 
performance  and  learning,  in  order  to  make 
adjustments.  Classroom  assessment  activities 
become  the  central  vehicle  for  clarifying 
criteria  to  let  students  (and  outsiders)  know 
what  counts,  and  to  direct  their  learning.” 
(Earl  & Cousins,  1995,  p.  44) 

Children  progress  through  observable 
stages  in  becoming  readers,  writers,  and 
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specific  outcomes  with  indicators  for  success 


thinkers,  and  teachers  are  astute  observers 
and  documented  of  the  visible  indicators 
of  students’  progress.  When  they  talk 
about  what  they  know  about  the  stages 
and  milestones  of  student  learning,  they 
clarify  criteria  about  what  counts,  and  direct 
learning  and  their  own  teaching.  When 
they  talk  about  what  a student  is  just 
beginning  to  or  what’s  coming  along  well, 
what’s  there  consistently  or  what’s  absolutely 
wonderful,  they  are  talking  about  what’s  not 
yet  addressed  (but  might  be),  what’s  progress- 
ing well  toward  standards  (and  needs  rein- 
forcement), what’s  meeting  standards  (and 
worthy  of  an  informal  report  to  student 
and  parents),  and  what’s  exceeding  standards 
(and  deserving  of  opportunity  and 
encouragement  for  further  excellence). 

In  conversation  with  each  other  over 
student  work  samples,  teachers  express 
extraordinarily  similar  observations  of  stu- 
dent growth,  albeit  sometimes  in  different 
language,  and  accompany  each  judgment 
either  overtly  or  implicitly  with  directions 
for  next  steps  in  teaching  that  child. 

Innovations  in  classroom  program 
delivery  in  the  1980s  and  1990s  broad- 
ened program  objectives,  content,  and 
assessment.  The  assessment  repertoire  of 
teachers  now  includes  a large  array  of 
assessment  methods: 

• teachers’  and  students’  anecdotal  records 

• journals  and  logs 

• checklists,  rating  scales 

• demonstrations  of  actual  performance 
of  tasks 

• independent  research  projects 

• student  self-evaluation  and  peer 
evaluation 

• portfolios,  simulation,  video,  audio  and 
photographed  records  of  achievement 
Description  of  this  broad  array  of 

program  and  assessment  methods  has 
involved  a huge  expansion  of  the  language 
used  to  describe  and  communicate  about 
student  progress  to  students  and  parents, 
not  standard  and  not  always  consistent 
irom  one  province,  district,  school,  grade, 
or  even  from  one  classroom  to  another. 
Teachers  may  observe  similar  features  of 
students’  academic  development,  but 


teacher  talk  reveals  a variety  of  words  used 
to  describe  what  they  see.  Parents  have 
expressed  dismay  that  they  don’t  under- 
stand the  messages  about  what  their  chil- 
dren should  know  and  be  able  to  do,  and 
how  well  they  are  doing  so.  Some  have 
advocated  a return  to  “basics”  and  the 
methods  and  language  they  remember 
from  their  own  school  days.  The  econom- 
ic, social,  and  political  tides  of  the  ’90s 
have  focused  increasing  public  attention 
on  student  achievement  levels  and  indica- 
tors of  progress.  It  is  important,  then,  that 
the  knowledge  teachers  have  about  stu- 
dents’ progress  be  expressed  in  a common 
language  for  clear  and  consistent  commu- 
nication with  students,  parents,  and  the 
public  about  individual  and  collective 
student  achievement.  When  teachers  talk 
about  what  teachers  know  about  chil- 
dren’s learning,  important  consensus 
emerges  about  markers  of  development 
and  avenues  for  instruction  — the  “stuff” 
of  student  and  school  improvement  — 
and  about  common  ways  of  describing 
these  to  students  and  parents. 

Aspects  of  student  performance  which 
lend  themselves  to  self-referenced  assess- 
ment are  relevant  to  all  grades  as  they  are 
developmental  in  nature,  increasing  and 
expanding  with  students’  age.  Development 
of  assessment  in  these  areas,  such  as  writ- 
ing skills,  can  provide  useful  classroom 
and  school  assessment  tools,  and  a forum 
for  important  teacher  talk  to  clarify  and 
make  consistent  the  criteria  of  assessment 
and  the  language  for  communicating 
about  progress. 

An  Exemplary  Assessment  Tool 

In  Hamilton,  a holistic  continuum  of 
growth  in  students’  writing  has  been 
developed,  reflecting  notable  shifts  in 
information,  organization,  and  presenta- 
tion from  Grade  1 to  Grade  8 (GO: 
Writing!  manuscript  in  preparation). We 
have  found  that  the  knowledge  of  teach- 
ers about  the  continuum  can  be  repre- 
sented by  25  stages.  The  dominant  features 
of  each  stage  are  described  in  words  and 
illustrated  by  student  writing  samples. 


Information  from  pilot  schools  indi- 
cates that  teachers  find  tins  assessment 
tool  useful  in  the  classroom  and  in 
reporting  to  parents.  They  are  able  to  use 
this  tool  to  assess  students  reliably  across 
teachers  and  consecutive  grades.  Such 
assessments  allow  for  self-referenced 
profiles  of  students’  progress  across  grades, 
which  can  also  be  evaluated  with  refer- 
ence to  a group  (grade)  or  standard.  They 
complement  group-referenced  assessments 
and  standards-referenced  reviews  and 
report  cards,  which  are  tagged  to  expecta- 
tions within  grades,  by  displaying  stu- 
dents’ learning  through  their  school  years 
with  reference  to  their  past  accomplish- 
ments and  next  steps. 

Teacher  collaboration  on  classroom 
assessment,  within  and  across  grades,  has 
great  potential  for  contributing  to  success 
of  school  improvement  — improvement 
m what  all  students  know  and  are  able 
to  do.  Classroom  assessment  will  always 
be  more  extensive  and  multi-faceted  than 
the  portions  which  can  be  shared  and 
summarized  for  school  planning.  However, 
shared  classroom  assessment  can  provide 
common  strands  of  information  across 
classrooms  and  grades  about  important 
outcomes  for  students,  for  specific  and 
frequent  feedback  about  the  success  ol 
school  plan  initiatives.  A visible  continuum 
of  examples  of  student  work  through  the 
grades  is  a powerful  educational  tool  tor 
the  entire  school  community,  displaying 
individual  and  collective  student  progress 
m the  complex  learning  and  thinking  skills 
required  by  students  of  the  ’90’s.  El 
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In  this  article,  we  explore  how  action 
research  can  foster  and  sustain  profes- 
sional community  in  schools  and  lead  to 
an  enriched  model  of  collegial  account- 
ability. We  begin  by  briefly  describing 
fundamental  issues  related  to  professional 
community,  accountability,  and  action 
research.  We  use  the  term  “professional 
communities  of  inquiry”  to  capture  the 
synergistic  relationship  between  these 
concepts.  Then  we  describe  a particular 
program  in  the  York  Region  Board 
of  Education  intended  to  support 
collaborative  action  research. 

Professional  Community 

We  start  with  the  premise  that  the  basis 
for  the  meaningful  exercise  of  responsibil- 
ity and  commitment  to  action  occurs 
when  there  is  (a)  a shared  responsibility 
for  student  learning  and  (b)  teachers  are 
confident  in  their  own  professional  judg- 
ments and  how  to  act  upon  them. 
Professional  community  provides  one 
means  of  increasing  shared  responsibility 
while  action  research  can  serve  as  an 
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effective  method  of  building  teachers’ 
confidence  in  their  own  judgments.  Yet,  it 
is  the  interplay  between  these  two 
approaches  that  can  really  enrich  teachers’ 
efforts  to  be  accountable.  Before  we 
examine  a case  study  that  demonstrates 
this  interplay,  we  need  to  examine  funda- 
mental qualities  of  both  professional 
community  and  action  research. 

Professional  community  can  be  defined 
as  “school  staff  members  taking  collective 
responsibility  for  achieving  a shared  edu- 
cational purpose,  and  collaborating  with 
one  another  to  achieve  that  purpose” 
(Newmann,  1994,  p.  1). When  teachers 
take  advantage  of  opportunities  to  collab- 
orate, they  are  able  to  work  toward  estab- 
lishing common,  clear,  and  consistent 
messages  about  expectations  of  learning. 
Furthermore,  their  influences  on  students 
are  more  likely  to  be  interdependent, 
powerful  and  prevalent.  As  a result,  when 
teachers  develop  a sense  of  community, 
personal  commitments  are  more  likely  to 
be  sustained  by  collective  responsibility. 

Unfortunately  there  are  few  organiza- 


ools 


tional  structures  that  help  teachers  to  collec- 
tively examine  student  progress  and  to  dis- 
cuss problems,  remedies,  and  consequences. 
In  order  to  facilitate  this  objective,  strategies 
are  required  that  provide  a focused  frame- 
work for  the  professional  community  to 
flourish  in  a meaningful  and  coherent  way. 

The  Promise  of  Action  Research 

Collaborative  action  research  provides 
one  strategy  for  effectively  organizing  and 
focusing  teachers’  efforts  to  develop  a 
professional  community.  Action  research 
“involves  systematic  inquiry,  either  indi- 
vidually or  collaboratively,  into  a self-iden- 
tified teaching/learning  problem  to  better 
understand  its  dynamics  and  to  provide 
follow-up  strategies  geared  towards  the 
problem’s  improvement”  (Hamilton,  1996, 
p. 4). This  inquiry  process  becomes  much 
more  meaningful  when  participants  collec- 
tively pose  questions,  reflect  together,  plan 
interventions,  and  develop  shared  solutions 
(Louis,  Kruse,  & Raywid,  1996). 

There  are  many  opportunities  within 
the  process  of  collaborative  inquiry  that 
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have  the  potential  to  strengthen  profes- 
sional community.  As  teachers  begin  to 
define  the  “problem”  being  investigated 
through  action  research,  a dialogue  about 
the  nature  of  the  issue  and  how  to  solve 
it  often  typifies  the  planning  process. 
Discussions,  exchanges  of  opinion,  and  the 
relaying  of  past  experiences  enrich  this 
meaning-making  process.  Fundamental 
questions  are  posed  and  answered  often  in 
dialectical  fashion.  Why  are  we  doing  this? 
What’s  the  issue  we ’re  investigating?  What 
do  we  want  to  achieve  by  doing  this? 

How  will  our  students  benefit?  What  do 
we  need  to  do  to  help  them?  How  are  we 
going  to  do  it?  Where  are  we  going  to 
find  the  time?  The  exchange  of  beliefs  and 
perceptions  that  accompany  this  often 
challenging  task  of  planning  and  preparing 
for  the  research  is  a critical  stage  in 
the  development  of  any  professional 
community  of  inquiry. 

This  collective  search  for  shared  mean- 
ing and  action  serves  as  a highly  effective 
way  of  ensuring  that  our  two  cornerstones 
of  accountability  — responsibility  and 
commitment  — are  firmly  situated.  When 
teachers  can  base  key  instructional 
decisions  on  concrete  mutually-derived 
evidence  of  success  — especially  if  the 
merit  and  the  nature  of  the  evidence  are 
negotiated  collectively  — they  will  possess 
more  confidence  that  the  follow-up  action 
will  be  successful.  When  one  teacher 
negotiates  the  nature  and  rationale  for  that 
action  with  other  informed  professionals, 
there  is  more  likelihood  that  a collective 
responsibility  will  ensue. 

Outcome-Based  Learning  Action 
Research  Project 

Over  the  last  three  years,  members  of 
Curriculum  and  Instructional  Services  in 
the  York  Region  Board  have  provided 
various  means  of  supporting  teachers  as 
they  implement  Outcome-Based  Learning 
in  their  classrooms  such  as  study  groups, 
inservice  sessions,  and  summer  professional 
development  institutes.  In  1996,  the 
OBL  Implementation  Plan  also  included 
support  for  a small  group  of  teachers  and 


administrators  to  study,  via  action  research, 
the  relationship  between  their  own  OBL- 
related  teaching  practices  and  improved 
student  learning. 

The  facilitators  of  the  program  recog- 
nized that  engaging  in  action  research  was 
a new  process  for  many  teachers.  In  fact, 
even  considering  the  word  “research”  was 
intimidating  for  many  people  who  had 
had  limited  involvement  in  systematic 
classroom-based  inquiry.  Thus,  the  facilita- 
tion team  provided  the  following  work- 
shops and  sessions  to  the  small  group  of 
teachers  and  administrators  involved  in  the 
OBL  Action  Research  Project: 

• advice  in  focusing  their  research  on  a 
specific  topic  and  preparing  action 
plans 

• workshops  to  encourage  the  develop- 
ment of  specific  inquiry-oriented  skills 

• resources  to  assist  the  participants  in 
learning  further  about  specific  aspects 
of  their  projects 

• sharing  sessions  to  encourage  project 
participants  to  network  and  learn  from 
one  another 

• limited  start-up  funds  that  were  used 
for  release  time,  supplies,  typing,  tran- 
scription, printing,  or  photocopying 
Although  not  specifically  required, 

applicants  were  encouraged  to  develop 
collaborative  processes  of  inquiry  wherever 
possible.  Eleven  projects  were  supported  in 
this  first  year  of  the  program.  All  of  them 
involved  some  degree  of  collaboration, 
whether  it  existed  between  teachers, 
teachers-students,  administrators-teachers, 
parents-teachers,  or  some  combination  of 
these  relationships. 

One  of  these  projects  is  highlighted  in 
this  article  because  it  featured  collaborative 
inquiry  on  many  different  levels  and, 
thereby,  exemplified  a dynamic  model  of 
professional  community  supported 
through  action  research.  The  following 
account  of  this  project  provides  many 
glimpses  into  the  benefits  and  challenges 
experienced  by  the  participating  teachers. 

The  Doncrest  Experience 

Five  special  educators  and  five  regular 


educators  at  Doncrest  Public  School 
focused  their  efforts  on  developing  a 
collaborative  team  approach  to  action 
research  during  the  1996/97  school  year. 
As  well  as  exploring  a problematic  issue  in 
their  school,  it  was  their  intent  to  learn 
more  about  their  own  active  engagement 
as  co-learners  in  this  inquiry  process  and 
what  it  means  to  be  teachers  who  research 
their  own  practices. 

Their  current  concern  or  challenge  was 
in  determining  how  to  use  an  exceptional 
student’s  Special  Education  Program  Plan 
(SEPP)  to  improve  the  working  relation- 
ship between  regular  and  special  educators 
in  their  school.  The  ultimate  goal  was 
to  ensure  that  the  exceptional  students 
receive  an  outcomes-based  learning 
program  that  could  be  implemented  and 
tracked  by  both  regular  educators  and 
special  educators.  Other  related  questions 
that  emerged  from  their  primary  question 
were:  How  can  we  reach  a more  equitable 
sharing  of  responsibility  for  an  exceptional 
student?  How  do  we  as  regular  and  special 
educators  define  “sharing  a student”?  Past 
practices  — from  their  own  school  as 
well  as  others  — demonstrated  a lack  of 
sharing  of  the  SEPP  between  special  and 
regular  educators  in  terms  of  the  develop- 
ment, implementation,  and  assessment  of 
the  exceptional  learner’s  goals.  In  reflecting 
on  these  experiences,  team  members 
believed  that  there  did  not  appear  to  have 
been  much  connection  made  between 
these  goals/outcomes  and  the  learning 
outcomes  outlined  in  the  regular  educa- 
tion classrooms. 

One  of  the  significant  ways  of  encour- 
aging collegiality  was  to  use  the  data 
collection  process  as  a means  of  promoting 
the  sharing  of  perspectives  and  developing 
a common  base  of  understanding.  It  began 
with  a pre-assessment  survey  developed  by 
several  team  members  that  asked  all  of  the 
ten  participants  to  examine  and  reflect 
upon  their  past  and  current  knowledge  of 
and  practices  with  the  SEPP. 

According  to  Louis,  Kruse,  and  Raywid 
(1996),  members  of  a professional  commu- 
nity hold  high  esteem  for  the  knowledge 
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they  jointly  create  and  find  ways  to 
generate  this  knowledge.  In  the  Doncrest 
project,  the  participants  believed  that  they 
had  expertise  at  their  fingertips.  Project 
participants  felt  it  was  more  instrumental 
to  better  use  the  valuable  human  resources 
they  possessed  in  their  own  school,  rather 
than  relying  on  sources  of  expertise 
beyond  its  walls.  The  key  was  in  using  the 
action  research  process  to  uncover  and 
legitimize  this  expertise.  Giangreco, 
Edelman,  and  Cloniger  (1993)  believe  that 
the  goals  and  objectives  set  out  in  the 
exceptional  learner’s  SEPP  or  individual 
education  plan  (IEP)  can  be  partially 
or  wholly  met  in  the  regular  education 
classroom  even  if  they  are  quite  different 
from  the  goals  or  learning  outcomes  of 
the  rest  of  the  class.  Shared  goals  and 
instruction  of  all  the  specialists  involved, 
however,  needed  to  be  established.  The 
project  team’s  pre-assessment  survey  helped 
them  to  discover  each  participant’s  particu- 
lar perspective.  From  this  data,  the  project 
team  could  identify  the  common  base  as 
well  as  the  gaps  in  understanding  between 
the  10  participants. 

Broad  based  themes  emerged  from  this 
collection  of  data  which  were  subsequent- 
ly shared  with  the  whole  team.  There  were 
four  main  contextual  factors  mentioned 
repeatedly  by  team  members: 

1 . Regular  and  special  educators  need 
to  provide  both  direct  support  in  the  class- 
room for  one  another  and  their  learners  as 
well  as  on-going  consultative  support  and 
other  resources  when  working  together  to 
create,  deliver  and  track  the  development 
of  the  exceptional  learner. 

2.  Special  educators  possess  certain 
knowledge  and  skills  that  need  to  be 
shared  with  regular  educators  who  are 
integrating  exceptional  learners  into  their 
classrooms. 

3.  A lack  of  clarity  existed  regarding 
roles  and  responsibilities  of  the  regular 
educator  and  special  educator  for  the 
exceptional  learner  within  the  context  of 
the  regular  education  classroom  and  the 
outcomes-based  approach  to  learning. 

4.  Time  was  a significant  limitation  in 


fostering  the  consistent  development  of 
collaborative  partnerships  for  the  excep- 
tional learner’s  program  development, 
implementation,  and  tracking. 

The  next  part  of  the  data  collection 
process  was  the  collection  and  collation  of 
exceptional  students’  writing  samples  over 
a four-month  period.  A regular  educator 
partnered  with  a special  educator  to  col- 
lect a monthly  writing  sample(s)  from  two 
exceptional  learners  as  a means  of  tracking 
their  written  language  development.  The 
partners  kept  a monthly  communication 
record  of  their  collaborative  efforts  includ- 
ing descriptions  of  their  conversations  and 
actions  surrounding  the  written  language 
programming  needs  of  the  two  exception- 
al learners.  In  total,  10  educators  were 
tracking  20  students. 

At  the  end  of  the  four  months,  the  sec- 
ond author  asked  each  teacher  to  submit  a 
journal  response  to  the  following  question: 
“Do  you  have  observable  evidence  that 
what  you’re  doing  is  facilitating  (a)  student 
growth  in  terms  of  written  language 
development  and  (b)  teacher  growth  in 
terms  of  the  communication  between  you 
and  your  partner?” 

Responses  from  the  special  education- 
classroom  teacher  partners  indicated  that 
the  writing  samples  and  other  evidence 
was  helpful  in  tracking  students’  develop- 
ment. Nevertheless,  the  degree  to  which 
the  partners  could  establish  a trusting  and 
mutually  supportive  relationship  influ- 
enced their  ability  to  use  this  data  to  joint- 
ly plan  for  the  students  and  observe  their 
growth.  It  was  clearly  evident  from  the 
submissions,  however,  that  even  for  those 
partnerships  that  were  at  more  formative 
stages,  the  collaborative  discussions  about 
issues  of  shared  responsibility  proved  to  be 
valuable  in  promoting  more  awareness  of 
the  potential  use  of  the  SEPP  and  helpful 
programming  strategies. 

From  this  account,  it  appeared  that  the 
action  research  process  had  provided  the 
initial  structural  support  for  the  emerging 
sense  of  professional  community.  From 
the  journal  entries,  it  was  evident  that  all 
members  of  the  team  clearly  saw  a need  to 


improve  communication  between  regular 
and  special  educators  so  that  a better 
understanding  of  the  exceptional  student, 
his/her  SEPP  and  a sharing  of  responsibil- 
ity for  both  could  be  developed.  All 
members  of  the  team  saw  this  action 
research  project  as  an  opportunity  to  work 
with  a group  of  teachers  who  recognized 
the  value  in  supporting  each  other’s 
attempts  to  improve  their  practices.  As  a 
team,  they  were  focusing  on  individual 
students’  learning  processes  in  the  context 
of  their  writing.  The  team  members  were 
looking  for  students’  growth  over  time  as 
outlined  in  their  Special  Education 
Program  Plans.  As  partners,  they  were 
willing  to  engage  in  the  observation  and 
documentation  of  individual  growth  in 
writing.  Nevertheless,  for  many  of  the 
participants,  it  was  the  first  time  they  had 
shared  concrete  observations,  assumptions, 
and  strategies  with  another  colleague 
about  a students’  progress. 

The  initial  engagement  among  team 
members  eventually  led  to  differences 
in  role  definitions  and  functions  which 
reinforced  the  further  entrenchment  of  a 
professional  community.  They  all  became 
both  regular  and  special  educators  as  they 
began  to  think  in  more  non-standard  or 
non-traditional  ways  about  their  positions 
in  the  school.  They  began  to  see  how  they 
might  redistribute  their  job  functions 
through  the  better  use  of  themselves  as 
human  resources.  They  began  to  refine  and 
redefine  their  roles.  They  became  more 
aware  how  fluid  these  role  definitions  were 
because  they  were  determined  by  the 
needs  and  skills  of  the  students  as  well  as 
their  own  needs  and  skills. 

In  reflecting  on  the  collective  experi- 
ences of  the  team,  three  factors  appeared 
to  help  to  make  the  relationship  between 
action  research  and  professional  communi- 
ty more  synergistic  for  the  Doncrest  team. 

First,  the  participants  had  to  view 
themselves  as  learners.  They  saw  this  action 
research  project  as  an  opportunity  to  work 
with  a team  of  teachers  who  would  value 
supporting  one  another  in  their  attempts 
to  improve  their  practices.  Simply  learning 
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to  ask  their  own  questions  and  be  responsi- 
ble for  following  through  with  an  inquiry, 
was  in  itself,  already  a valuable  outcome  ot 
their  action  research  project. 

Second,  some  participants  must  be 
prepared  to  assume  leadership  for  coordi- 
nating the  intentional  effort  to  keep  mov- 
ing the  process  forward.  For  the  Doncrest 
project,  the  second  author  assumed  this 
responsibility.  Engaging  in  this  particular 
collaborative  initiative  complicated  the 
issue  of  her  role  in  this  project.  Not  only 
was  she  a partner  in  such  a team  inquiry, 
she  was  also  the  coordinator,  coach,  and 
confidante.  She  soon  found  that  she  had  to 
focus  not  only  on  what  was  happening 
in  terms  of  the  overall  team  growth,  an 
individual  teacher’s  growth,  the  students’ 
growth,  but  also  on  what  was  happening 
to  her  as  an  educational  leader  of  the  team 
and  how  she  could  continue  to  best  sup- 
port all  ot  the  team  members. 

Third,  team  members  had  to  demon- 
strate patience  with  negotiating  meaning 
and  action  throughout  the  process.  The 
team  members  valued  the  dialogue  that 
occurred  as  they  focused  their  discussions 
during  the  on-going  informal  partner 
meetings  and  monthly  formal  whole  group 
meetings  on  using  the  SEPP  as  a commu- 
nication tool  to  improve  students’  written 
language  development.  All  members  active- 
ly engaged  in  a collaborative  construction 
of  knowledge  that  allowed  each  individual 
to  contribute  perspectives  and  ideas  to 
the  inquiry.  Conflict  or  dissonance  did 
occur  on  occasion  when  a colleague’s 
expectations  appeared  inconsistent  with 
another  colleague’s  sense  of  reality  of  her 
own  classroom. 

Fourth,  patience  and  persistence  have  to 
be  coupled  with  respect  for  diversity  of 
practices  and  growth  experiences.  According 
to  the  second  author,  participants  perceived 
that  coming  to  a shared  understanding  of 
the  action  research  process  included  under- 
standing that  individual  members  of  the 
team  were  at  different  stages  in  their  devel- 
opment regarding  (a)  communication  and 
collaboration  with  their  teaching  partner 
and  (b)  their  understanding  of  how  shared 


responsibility  for  exceptional  learners  trans- 
lates into  classroom  practices.  They  learned 
that  this  process  is  perceived  in  different 
ways  by  different  members  of  their  team. 
They  learned  that  this  process  has  not  nec- 
essarily contributed  to  a sustained  improved 
working  relationship  between  all  regular 
and  special  education  teachers  on  the  team. 
However,  all  members  have  asked  them- 
selves why  or  why  not  it  has  occurred. 
According  to  the  team,  this  was  positive 
outcome  by  itself. 

H ow  did  this  synergy  lead  to  a more 
authentic  and  meaningful  form  ot  account- 
ability? As  a team,  the  partners  in  the 
Doncrest  project  believed  that  their 
dialogue,  reflection  and  actions  from 
November  1996  through  April  1997  have 
brought  about  significant  student  and 
teacher  improvement  and  a greater  sense 
of  shared  responsibility  for  student  achieve- 
ment. Each  partnership  experienced  a dif- 
ferent degree  of  success  and  measured  it  in 
different  terms.  As  a team,  however,  they 
believed  that  they  were  successful  in  their 
attempt  to  engage  in  a systematic  inquiry 
for  the  first  time  where  they  examined 
what  they  were  doing  as  teachers,  how 
they  were  doing  it  and  how  they  might  do 
it  differently  the  next  time.  As  a team,  they 
were  successful  in  attempting  to  define  the 
problem  more  clearly  and  in  developing 
some  strategies  that  facilitated  its  improve- 
ment. Consequently,  all  members  actively 
sought  to  improve  communication 
between  their  partners  so  that  a better 
understanding  of  the  exceptional  student, 
his/her  SEPP  and  a sharing  of  responsibili- 
ty for  both  could  be  developed. 

What  lies  ahead  for  the  Doncrest 
project?  Participants  are  willing  to  contin- 
ue to  examine  assumptions  about  teaching 
and  learning.  They  are  cognizant  that  their 
initial  inquiry  has  led  to  further  related 
inquiries.  The  success  of  their  first  year  in 
the  action  research  project  lies  in  their 
shared  responsibility  and  sustained  commit- 
ment to  continue  to  build  their  relation- 
ships with  their  colleagues  and  with  their 
exceptional  students  and  with  the  recogni- 
tion that  ongoing  analysis  of  both  their 


own  learning  and  student  learning  is  a 
significant  form  of  accountability  in  itself. 
This  type  of  interaction,  inquiry,  and 
analysis  is  essential  for  continuous  teacher, 
student  and  school  improvement. 

Final  Reflections 

The  Doncrest  experience  provided  a 
unique  framework  for  exploring  the  rela- 
tionship between  professional  community 
and  action  research.  By  examining  this 
interplay,  we  have  learned  that  professional 
community  is  not  a static  entity  but  an 
active  construction  of  experience  and  out- 
comes that  is  constantly  and  dynamically 
evolving.  Developing  a structured  program 
of  inquiry  through  action  research  served 
to  center  the  community-building  efforts 
and  experiences  on  substantive  and  endur- 
ing issues  of  practice. 

Developing  a professional  community 
also  has  implications  for  supporting  effec- 
tive and  meaningful  action  research.  Inquiry 
becomes  more  powerful  and  enriching 
when  teachers  work  towards  developing  a 
shared  understanding  of  the  definitions  of 
the  problem,  the  means  of  investigating  it, 
the  salient  aspects  of  the  findings,  and  the 
action  required  to  resolve  it.  When  teachers 
inquire  together,  they  think,  talk,  reflect, 
and  act  together.  What  child  would  not 
benefit  from  this  shared  commitment?  131 
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Professional  Learning  Update 


BUILDING  A CAPACITY  FOR  CHANGE 


Secondary  School  Renewal 


A Learning  Opportunity  Presented  by  OISE/UT  in  co-operation  with  the  Ministry  of  Education  & Training 


Beaver  Brae  Renewal  Team  (above) 

Thirteen  teams  of  Ontario  teachers,  department  heads,  and  principals 
who  took  part  in  the  Summer  Institute  on  change  and  best  practices  in 
secondary  schools  are  now  in  the  midst  of  Implementation  Projects.  These 
projects,  conceived  by  the  team  as  a way  to  stimulate  renewal  at  their 
school,  will  be  reviewed  and  refined  at  a follow-up  session  at  OISE/UT 
in  the  spring  of  ’98.  Three  of  the  schools  report  on  their  projects. 


FROM  NORTH  YORK  Northview  Secondary  School: 

“When  we  returned  to  school  in  the  fall,  we  began  our  exercise  in 
curriculum  renewal  by  asking,  what  should  Northview  graduates 
get  from  their  education?  A list  of  over  75  skills  was  produced  by 
Department  Heads  and  Staff  Leaders.  When  we  studied  the  skills,  we 
noted  three-quarters  were  Relationship  Skills  (ranging  from  emotion- 
al growth  and  sense  of  personal  identity  to  healthy  lifestyle,  openness 
to  new  ideas  and  awareness  of  others).  The  rest  of  the  skills  listed 
could  be  categorized  as  Cognitive  or  Lifelong  Learning  Skills  (ranging 
from  literacy  competencies  to  technological  comfort  to  organizational 
marketing  abilities). Usually  these  skills  are  not  the  initial  focus,  but 
an  unpredictable  by-product  of  our  teaching.  Once  we  refined  our  list 
a bit  more,  we  invited  all  our  Staff  Leaders  to  choose  one  skill,  and 
then  answer  some  very  pragmatic  questions  about  that  skill,  including 
‘What  does  this  skill  look  like?’  and  ‘How  would  I use  my  subject 
discipline  to  teach  it?’  We  are  now  ready  to  take  all  our  department 
members  through  this  same  exercise  and  then  develop  a school  plan 
and  philosophy  based  on  critical  skills  for  Northview  graduates.  The 
general  staff  exercise  was  to  have  occurred  at  the  end  of  October 
but  was  eliminated  by  the  Teacher  Job  Action.  Hopefully  we  will  be 
able  to  do  this  in  team  and  department  meetings  over  December 
and  January.  We  have  been  slowed  down,  but  not  stopped!” 


FROM  BROCKVTLLE  South  Grenville  District  High  School: 


“The  Capacity  for  Change  team  at  South  Grenville  District  High 
School  has  been  busy  during  the  first  two  months  of  the  new  school 
year  bringing  the  lessons  learned  at  the  Summer  Institute  to  the 
staff  at  SGDHS.  We  presented  an  overview  of  the  Summer  Institute 
to  our  leadership  team  and  our  Director  of  Education  during  the 
last  week  of  August.  Having  gained  their  enthusiastic  support,  several 
activities  were  carried  out  on  the  September  P.A.  Day  to  involve 
our  staff  in  creating  a vision  for  our  school  which  would  authenti- 
cally meet  the  needs  of  our  student  population.  That  work  was 
followed  up  at  the  October  P.A.  Day  with  two  workshops:  one 
which  showed  our  staff  how  to  combine  co-operative  learning  with 
computer-based  learning  activities  and  one  which  brought  represen- 
tatives from  local  business  and  industry  to  dialogue  with  us  about 
their  requirements  for  our  graduates  as  they  enter  the  work  force. 
Immediately  following  our  P.A.  Day  of  October  24,  we  entered 
into  the  province-wide  Job  Action  which  has  now  resulted  in  the 
cancellation  of  any  future  P.A.  Days  in  our  jurisdiction.  Our  task 
as  a Capacity  for  Change  team  will  be  to  keep  the  momentum  of 
the  first  two  months  and  carry  out  other  plans  for  staff  development 
during  the  remainder  of  the  school  year  in  a ‘unique’  climate  for 
staff  development!” 


FROM  KENORA  Beaver  Brae  Secondary  School: 


“We  returned  from  the  Summer  Institute  at  OISE/UT  with  a 
great  deal  of  enthusiasm  for  renewal.  The  excellent  presentations 
and  ensuing  discussion  provided  us  with  a wealth  of  possibilities 
for  enhancing  what  we  offer  to  the  students  of  Beaver  Brae. 

After  many  meetings  and  much  debate,  we  chose  to  focus  on 
developing  career  portfolios.  We  have  purchased  resources  from 
various  boards,  attended  a number  of  workshops  on  portfolios 
and  career  education  and  purchased  the  Choices  career  software. 

We  are  presently  analyzing  the  various  portfolio  formats  and 
delivery  methods.  We  will  be  piloting  several  of  these  in  the  second 
semester  and  have  invited  any  interested  staff  to  be  partners  in 
this  process.  After  reviewing  the  pilots,  our  plan  is  to  have  career 
planning  portfolios  in  place  for  Grade  9’s  for  the  fall  of  ’98.  There 
is  a great  deal  of  enthusiasm  among  the  team  and  the  staff  for 
the  project.” 


Orbit  will  keep  you  posted  on  this  unique  professional  learning 
opportunity  as  it  evolves  over  the  year. 


A Divergent 


Viewpoint 

Assessment  n i - * ■ i 

for  Political  4 

Accountability  or  School  Improvement?) 


Malcolm  A.  Levin 

Professor  Emeritus, 
OISE/UT 


Last  spring,  Grade  3 students  in  the  province  of 
Ontario  were  subjected  to  a system- wide  assess- 
ment of  their  language  and  mathematical  knowl- 
edge. The  testing  took  the  better  part  of  two  weeks 
and  was  reported  to  have  cost  over  $7  million.  It 
was  designed  and  conducted  under  the  auspices  of 
an  “arms-length”  agency  created  and  funded  by  the  Ministry 
of  Education  and  Training  following  recommendations  of  the 
1994  report  of  the  Royal  Commission  on  Learning.  This  assess- 
ment, which  is  to  take  place  annually,  and  the  Office  which 
carries  it  out,  were  put  in  place  ostensibly  to  serve  two  purposes: 

(1)  to  make  schools  and  teachers  more  accountable  to  parents 
and  taxpayers  for  producing  specified  learning  outcomes;  and 

(2)  to  improve  the  quality  and  effectiveness  of  schooling  in  the 
province. 

In  this  brief  essay  I suggest  that  these  two  goals  may  work  at 
cross-purposes  as  they  are  grounded  in  a misunderstanding  of 
the  functions  and  limits  of  group  testing  and  in  a rather  naive 
view  of  the  politics  of  evaluation  and  accountability.  Although 
political  accountability  and  educational  improvement  exist 
side-by-side  in  the  school  system,  their  relationship  is  not  well 
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understood  by  those  who  believe  that  “higher  standards”  and 
stricter  accountability' for  achieving  specified  learning  outcomes 
will  improve  the  quality  of  education  experienced  by  students. 
Moreover,  the  creation  of  new  organizations  and  more  elaborate 
reporting  procedures  to  monitor  and  evaluate  the  performance  of 
schools,  teachers  and  students  places  an  additional  financial  bur- 
den on  school  boards  who  have  been  experiencing  the  pangs  of 
shrinking  operating  budgets  tor  several  years.  Most  of  the  profes- 
sionals in  the  provincial  ministries  dealing  with  education  in 
Ontario  over  the  past  three  decades  were  aware  of  this  problem 
and  resisted  past  demands  tor  province-wide  testing.  Instinctively 
they  knew  that  giving  in  to  these  demands  would  undermine 
new  program  initiatives  undertaken  by  boards  of  education  and 
teachers  to  meet  the  needs  of  an  increasingly  diverse  (ethnically 
and  linguistically)  school  population  and  to  retain  students  in  the 
system  until  graduation. 

Until  the  economic  downturn  of  the  ’90s,  accommodation  of 
diversity  and  accessibility  took  precedence  over  assessment  and 
strict  accountability.  But  the  economic  and  political  environment 
within  which  the  schools  operate  has  changed  dramatically  over 
the  past  decade  as  governments  of  all  political  parties  have 
become  obsessed  with  cutting  expenditures  to  reduce  operating 
deficits  in  a stagnant  economy  where  neo-conservative  free  mar- 
keteers demand  more  value  for  money  in  public  education.  It  is 
in  this  climate  of  financial  crisis,  economic  fundamentalism  and 
political  conservatism  that  we  find  educational  professionals  who 
should  know  better  joining  the  chorus  of  critics  demanding  bet- 
ter results  for  less  money.  Since  the  demands  are  coming  from 
their  political  overlords  who  control  the  purse  strings  we  should 
not  be  surprised  at  their  acquiescence. 

The  current  demand  for  province-wide  testing  of  student 
achievement  in  language  and  mathematics  (and  by  extension 
for  national  standards)  arises  from  concerns  expressed  by  parents, 
employers,  politicians  and  university  professors  about  alleged 
deficiencies  in  these  areas  among  graduates  of  our  elementary  and 
secondary  schools.  There  is  some  hard  evidence  to  support  these 
concerns  in  published  international  comparisons  of  standardized 
test  scores  in  mathematics  and  science  earned  by  Canadian 
students  in  relation  to  those  in  selected  countries  in  Europe  and 
Asia.  (Apparently  American  students  aren’t  doing  so  well  in  the 
competition  either  despite  the  prevalence  of  standardized  testing 
in  the  U.S.)  However,  most  of  the  evidence  for  these  claims 
is  impressionistic,  gleaned  from  the  critical  comments  of 
spokespersons  selected  by  the  media  who  assert  that  there  has 


been  a major  decline  in  literacy  and  numeracy  among  young 
people  over  the  past  decade  or  two  or  three,  depending  on  the 
length  of  their  memories. 

With  the  possible  exception  of  longitudinal  comparisons  of 
Scholastic  Aptitude  Test  (SAT)  scores  in  the  U.S.,  we  have 
little  evidence  on  which  to  either  accept  or  reject  these  claims. 
Given  the  propensity  of  conservative  critics  of  the  public  schools 
to  romanticize  the  past,  their  comparisons  must  be  viewed 
with  skepticism.  Comparisons  of  test  scores  over  time  must  also 
be  interpreted  with  caution  as  they  are  based  on  different  popula- 
tions with  different  demographic  characteristics.  Moreover, 
the  SAT’s  themselves  are  periodically  revised,  updated  and 
re-normed.  Like  well-designed  political  polls,  these  tests  may 
provide  a valid  snapshot  of  the  knowledge  of  a given  population 
at  one  point  in  time,  but  they  may  be  poor  predictors  of  future 
academic  performance. 

Notwithstanding  these  limitations,  it  would  be  unwise  to 
dismiss  out  of  hand  underlying  concerns  about  the  quality  of 
educational  experiences  of  children  in  our  public  schools, 
especially  as  class  sizes  grow  and  resources  for  teachers  shrink 
under  the  impact  of  budgetary  cutbacks.  Even  if  the  schools  are 
providing  learning  environments  that  are  demonstrably  superior 
to  what  our  parents  and  grandparents  enjoyed  in  the  past,  there  is 
still  a gap  between  our  aspirations  for  the  public  school  system 
and  current  realities.  Too  many  young  people  leave  the  school 
system,  before  or  after  earning  a diploma,  with  a diminished, 
rather  than  enhanced,  sense  of  their  intellectual  capacities, 
interests  and  options  in  life.  However,  it  is  not  at  all  clear  how 
stricter  monitoring  of  academic  achievement  and  failure  through 
a system  of  annual  provincial  (or  even  school  or  board-based) 
tests  will  contribute  to  improving  the  quality  of  children’s 
educational  experiences  in  schools.  It  may  even  have  the  opposite 
effect,  as  many  well-intentioned  bureaucratic  interventions 
often  do.  How  then  do  we  account  for  all  this  support  for 
mass  testing? 

To  begin,  I would  suggest  that  the  current  revival  of  educa- 
tional scientism  via  standardized  testing  has  a lot  more  to  do  with 
the  government’s  concern  for  its  own  image  as  overseer  of  the 
school  system  than  with  the  efforts  of  teachers  and  principals  to 
improve  the  quality  of  educational  experiences  of  the  young 
people  in  their  charge.  Education  may  be  a personal  quest  for 
knowledge,  understanding  and  self-discovery,  but  public  schools 
are  instruments  of  State  policy  and  are  accountable  to  the  State 
for  their  actions.  Historically,  mass  testing  has  proved  to  be  an 
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effective  instrument  of  indirect  control  over  what  teachers  and 
children  actually  do  in  their  classrooms.  Its  roots  lie  in  the  19th 
century  Scottish  system  where  government  inspectors  actually 
went  from  school  to  school  to  test  the  school  knowledge  of  the 
children  and  school  grants  were  adjusted  on  the  basis  of  the 
results  of  these  tests.  The  current  effort  is  a far  cry  from  this  but 
it  does  provide  visible  evidence  that  the  government  is  taking  its 
role  as  custodian  of  the  public  schools  seriously. 

A second  reason  for  the  current  revival  of  mass  testing  arises 
from  the  long-standing  practice  of  politicians,  businesspeople, 
universities  and  the  mass  media  of  blaming  the  public  schools  — 
and  by  extension,  teachers,  parents  and  kids  — for  nearly  every 
economic,  social  and  political  problem  in  our  society.  It  the 
economy  is  sluggish,  schools  are  criticized  for  not  instilling  the 
right  skills,  attitudes  and  motivation  in  their  students.  Perceived 
increases  in  crime,  violence,  use  of  illegal  drugs  and  broadcasting 
of  subversive  music  and  art  are  also  blamed  on  an  alleged  break- 
down of  “family  values”  at  home  and  “discipline”  in  the  schools. 
If  the  schools  are  not  seen  as  the  cause  of  these  problems  they  are 
certainly  taken  to  task  for  not  solving  them. 

Finally,  support  for  mass  testing  is  rooted  in  a widespread  pro- 
duction view  of  schooling  in  which  young  persons  are  conceived 
as  raw  material  to  be  manufactured  into  more  or  less  finished 
products  for  the  university  and/ or  labour  market.  While  the  mis- 
sion of  the  schools  is  multi-faceted  with  at  least  five  components 
— academic,  civic,  developmental,  social  and  vocational  — only 
one  of  these,  the  developmental,  takes  the  child  as  its  starting 
point.  The  others  begin  with  the  needs  and  requirements  of 
Academia,  the  State,  the  Community,  and  Business.  From  their 
perspective  the  task  of  the  schools  is  to  inculcate  in  children  the 
attitudes,  knowledge,  skills  and  values  that  will  transform  them 
from  egocentric  individuals  into  “responsible”  citizens,  consumers 
and  workers.  Flistorically,  the  schools  academic,  civic  and  voca- 
tional missions  have  dominated  the  agenda,  with  the  academic  at 
the  top.  However,  as  the  North  American  economy  has  shifted 
from  labour-intensive  to  technology-intensive  and  millions  of 
traditional  jobs  have  disappeared  or  have  been  down-graded  or 
exported,  the  central  preoccupation  of  school  critics  has  become 
its  vocational  mission.  The  current  demand  is  less  for  academic 
excellence  than  for  preparing  young  people  to  survive  and 
succeed  in  the  emerging  global  rat-race.  In  brief,  if  children  are 
viewed  as  raw  materials  and  schools  are  seen  as  turning  out  prod- 
ucts for  the  new  economy,  then  systematic  testing  of  the  products 
and  the  producers  in  the  name  of  quality  control  makes  sense.  It 
also  fits  rather  well  with  the  dismantling  of  the  welfare  state  and 
the  re-assertion  of  competitive  individualism. 

Demands  tor  mass  testing  of  students  and  stricter  accountabili- 
ty escalate  in  periods  of  economic  and  political  malaise,  especially 
when  government  funds  allocated  for  education,  social  services 
and  health  care  are  shrinking  while  demands  for  these  services  are 
growing.  In  these  times,  government  bureaucracies  at  all  levels  are 


under  pressure  from  the  media  and  opposition  politicians  to  justify 
their  budgets,  even  their  existence.  Since  the  core  mandate  of 
government  bureaucracies  is  to  regulate  public  services,  periodic 
evaluations  and  annual  reports  are  the  tools  commonly  used  to 
carry  out  this  mandate.  The  imposition  of  province-wide  testing 
and  a centralized  provincial  curriculum  not  only  highlights  the 
Ministry  of  Education  and  Training’s  regulatory  mandate,  but  also 
marks  the  end  of  three  decades  of  decentralization  of  Ontario’s 
school  system.  The  recently  introduced  legislation  regarding 
provincial  control  over  school  funding  and  further  consolidation 
of  school  boards  puts  the  finishing  touches  to  this  process. 

To  sum  up,  the  province  of  Ontario  is  moving  rapidly  to 
replace  a decentralized  system  of  schooling  in  which  meeting 
the  academic,  social,  and  developmental  needs  and  interests  of 
students  has  been  a major  concern  over  the  past  three  decades 
with  a centralized  system  dominated  by  a vocational/corporate 
agenda.  An  expanding  school  system  aimed  at  accommodating  an 
increasingly  diverse  population  of  students,  including  many  non- 
English  speakers,  and  preoccupied  with  issues  of  accessibility  and 
equity  is  being  supplanted  by  a contracting  system  obsessed  with 
accountability  in  the  face  of  growing  austerity  in  the  public 
service  sector.  The  revival  of  mass  testing  accompanies  the  resur- 
gence of  a factory  model  of  schooling  in  a job-scarce  competitive 
economy  in  which  graduates  must  sell  themselves  in  a labour 
market  dominated  by  emerging  new  information  technologies, 
chronic  re-organization,  and  lack  of  employment  security. 

Before  investing  a lot  of  time  and  scarce  financial  resources 
in  mass  testing  programs  governments  should  ask  what  educational 
benefits  can  be  reasonably  expected  from  such  an  investment. 
Based  on  the  American  experience  with  national  Scholastic 
Aptitude  Tests  written  by  high  school  students  seeking  admission 
to  colleges  and  universities  we  should  not  expect  much.  Average 
scores  on  these  tests  appear  to  be  declining  over  time  but  it  is 
impossible  to  determine  if  this  is  due  to  an  alleged  decline  in  the 
quality  of  high  school  programs  or  to  the  larger  number  and 
diversity  of  those  writing  the  exams  each  year.  In  any  case  if  mass 
testing  were  an  effective  tool  for  improving  school  programs  and 
instruction  we  might  expect  the  opposite  result.  Parenthetically, 
since  the  academic  credentials  and  grade  point  averages  of 
Canadian  teachers  (especially  in  the  elementary  panel)  have  risen 
significantly  in  recent  decades  as  competition  for  scarce  places  in 
our  faculties  of  education  restricts  entry  into  the  profession  to 
those  with  the  highest  academic  standing,  how  can  we  reasonably 
blame  the  alleged  failures  of  our  schools  to  solve  our  society’s 
economic  and  social  ills  on  teachers  who  are  winners  in  a much 
stiffer  academic  competition  than  existed  in  the  past? 

In  our  obsession  with  measuring  achievement  and  identifying 
problems  and  deficiencies  in  children  — and,  by  implication,  in 
their  teachers  and  parents  — we  are  neglecting  more  important 
issues.  What  can  we  do  and  what  are  we  willing  to  do  to  improve 
the  conditions,  (e.g.,  poverty,  hunger,  illness,  inadequately 
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equipped  and  crowded  classrooms)  that  give  rise  to  results  we  are 
so  unhappy  with?  Are  we  willing  to  put  more  human  and  material 
resources  into  ensuring  that  yqung  children  receive  the  care, 
support,  attention  and  . guidance  they  need  to  develop  fully  their 
individual  potentialities?  Are  we  prepared  to  put  more  teachers  in 
our  classrooms  and  to  provide  them  with  the  resources  and  sup- 
port that  they  need  to  fulfil  their  professional  commitments 
to  the  children  and  the  community?  Are  we  prepared  to  help 
parents  do  their  part  as  well?  Investing  more  resources  in  these 
areas  is  far  more  likely  to  improve  the  quality  and  effectiveness 
of  children’s  educational  experiences  in  schools  than  spending 
millions  on  testing  programs. 

The  typical  response  to  this  line  of  argument  is  that  “there  is 
no  money  to  do  this.”  Given  the  virtual  abandonment  by  federal 
and  provincial  governments  of  their  mandate  to  protect  the 
public  interest  and  to  ensure  that  all  Canadians  share  (to  different 
degrees)  in  this  country’s  vast  wealth  this  response  is  not  surpris- 
ing, absurd  though  it  seems.  A country  that  can  spend  billions 
on  gambling  in  casinos,  lotteries  and  on  the  stock  market,  and  on 
expensive  cars,  military  hardware,  prisons,  professional  sports,  huge 
shopping  and  entertainment  complexes,  and  designer  clothes  for 
children  as  well  as  adults  should  be  able  to  afford  to  provide  every 
child  with  a proper  start  in  life  — nutritionally,  medically  and 
educationally.  Recently,  Canada’s  national  newspaper.  The  Globe 
and  Mail  re-discovered  the  age-old  connection  between  poverty 
and  poor  school  performance  in  Canada.  Do  we  need  more 
studies  and  more  tests  to  tell  us  what  we  have  known  all  along? 

All  of  these  caveats  notwithstanding,  teachers  are  and  should 
be  concerned  about  monitoring  and  evaluating  the  progress  of 
their  students.  But  education,  unlike  schooling,  is  an  individual 
process,  and  progress  must  be  assessed  on  an  individual  basis. 
Teachers  know  this  in  their  daily  work  as  they  struggle,  under 
factory-like  conditions,  to  help  individual  students  achieve 
higher  levels  of  skill  and  understanding.  Imposing  competitive 
standardized  examinations  which  pit  students  against  each  other 
can  place  an  additional,  unnecessary  and  unfair  burden  on 
both  teachers  and  students.  Rather  than  improving  the  quality 
of  educational  life  in  classrooms,  it  can  undermine  it  as  both 
teachers  and  students  set  aside  such  “marginal”  classroom  activities 
as  art,  science,  health,  physical  education,  music,  French,  and  social 
studies  to  prepare  for  “high  stakes”  testing.  Is  this  what  we  want 
for  our  children  and  grandchildren? 

A number  of  Ontario  public  school  boards  have  attempted  to 
improve  the  quality  of  education  experienced  by  students  in  then- 
schools  through  a variety  of  “school  improvement”  projects  which 
attempt  to  utilize  systematic  assessment  data.  These  projects, 


described  elsewhere  in  this  special  Orbit  issue,  invite  administra- 
tors, teachers,  parents,  “community”  persons  and,  occasionally, 
students  to  express  their  views  and  make  recommendations  on  a 
range  of  matters  relating  to  school  performance  — e.g.,  academic 
achievement,  curriculum,  school  climate,  discipline,  school-com- 
munity relations,  and  so  on.  “Self-review”  seems  to  have  become 
the  preferred  model  but  the  “audience”  is  external  as  well  as  inter- 
nal. Some  of  these  projects  appear  to  have  stimulated  new  program 
initiatives  — e.g.,  breakfast  programs,  computer  labs,  collaboration 
with  local  businesses  to  enhance  the  learning  resources  available 
to  the  school  — which  seem  to  have  improved  the  quality  of  life 
in  the  schools  involved,  at  least  for  some  students  and  teachers. 

Almost  invariably,  participants  report  enthusiastically  that  their 
involvement  in  this  process  has  been  professionally  challenging 
and  stimulating  and  that  it  has  been  an  invaluable  personal 
learning  experience.  Typically,  they  are  less  clear  about  how  their 
work  has  affected  the  rest  of  the  teachers  and  students  in  the 
school  and  board.  They  come  by  this  uncertainty  honestly  for  the 
connections  between,  on  the  one  hand,  the  process  they  have 
engaged  in  and  any  documents  they  have  produced,  and,  on  the 
other  hand,  any  changes  in  the  day-to-day  operations  of  their 
schools  are  subtle,  complex  and  difficult  to  pin  down  — for 
researchers  as  well  as  practitioners. 

So  we  fall  back  on  process.  Visioning,  planning,  assessment,  col- 
laboration, team-work,  team-building  and  performance  monitoring 
properly  conducted  are  all  deemed  to  be  good  things  in  themselves 
even  if  we  cannot  clearly  demonstrate  how  they  improve  the  quali- 
ty of  educational  experience  for  students.  However,  they  are  rarely 
found  in  schools  except  when  special  projects  like  internal  program 
reviews  and  school  improvement  projects  require  them.  They  are 
not  found  because  the  organization  of  time  and  the  additional 
resources  needed  to  support  them  are  not  provided  unless  a special 
project  is  undertaken,  sometimes  not  even  then. Thus  any  organiza- 
tional benefits  of  these  projects  are  not  likely  to  survive  for  long 
after  the  project  is  over  and  things  return  to  normal.  In  the  end  the 
issue  comes  down  to  where  scarce  resources  should  be  allocated 
to  achieve  the  better  educational  experiences  we  all  want  for  our 
children  in  our  schools. 

In  this  respect,  school-based  efforts  that  support  teachers  in 
collaborative  efforts  to  create  more  stimulating  learning  environ- 
ments for  their  students  seem  a lot  more  worthy  of  support  than 
provincial  and  board  assessment  offices.  Province-wide  testing  of 
students  may  help  the  government  improve  its  image  as  overseer 
of  the  province’s  school  system  and  may  get  some  of  the  critics 
off  its  back.  Is  this  worth  the  expenditure  of  millions  of  dollars 
and  thousands  of  hours  of  valuable  teacher  and  student  time?  O 
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